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1 .  INTRODUCTION 


1.1  Scope 

This  document  constitutes  the  final  report  on  research  into 
"Reliability  Techniques  for  Combined  Hardware/Software  Systems," 
performed  for  the  U.S.  Air  Force  under  the  contractual  auspices 
of  Rome  Air  Development  Center,  Griffiss  Air  Force  Base,  Rome, 
New  York.  This  report  fully  satisfies  the  requirements  of  CLIN 
0002,  ELIN  A004,  of  contract  F3  06 02 -89 -C- 0111 ,  Reliability 
Techniques  for  Combined  Hardware  and  Software  Systems. 

1.2  Research.  Problem  Background 

Many  modern  military  and  commercial  systems  and  products  contain 
both  hardware  and  executable  software  elements.  The  hardware 
comprises  the  physical,  tangible  elements  of  the  system. 
Executable  software  is  a  logical  element  of  the  system, 
consisting  of  sequences  of  instructions  (programs) .  As  speed, 
capacity,  and  cost-effectiveness  advance,  an  ever-increasing 
number  of  applications  are  being  found  for  computers.  Society 
and  its  institutions  are  becoming  more  and  more  dependent  upon 
computers.  Microprocessors  are  being  embedded  into  all  manner  of 
devices  and  systems  to  add  "intelligence."  Computers  lie  at  the 
heart  of  real-time  applications  that  control  critical  and  vital 
functions,  where  system  failure  can  have  catastrophic  results. 
High  reliability  is  a  crucial  need  of  complex  military  systems, 
which  can  inflict  or  prevent  death.  As  hardware  becomes  more  and 
more  reliable  as  the  result  of  technological  advances,  software 
is  assuming  ah  increasing  role  in  achieving  system  reliability 
requirements.  System  reliability  considerations  have  in  many 
cases  made  a  transition  from  hardware  to  software. 

Methods  for  the  prediction  and  measurement  of  hardware 
reliability  are  considerably  older  and  more  mature  than  those  for 
software.  The  hardware  methods  have  become  standardized  and 
institutionalized  throughout  the  defense  and  related  industries 
through  military  standards  and  handbooks.  Over  the  past  fifteen 
years,  the  field  of  software  reliability  has  made  significant 
progress.  Major  achievements  have  taken  place  in  understanding 
software  reliability,  including  modeling,  prediction,  estimation, 
its  relationship  to  software  quality  metrics,  and  its 
relationship  to  mission  environment.  These  results  are  scattered 
throughout  the  industrial  and  academic  literature.  This  study  is 
intended  to  bring  this  research  together,  and  build  on  it,  adding 
now  theoretical  and  empirical  results  to  create  a  comprehensive 
methodology  for  software  reliability  assessment. 

The  methodology  resulting  from  this  study  covers  the  entire  life 
cycle,  is  compatible  with  the  military  standard  for  defense 
system  software  development — DOD-STD-2167A — and  is  aligned  with 
existing  hardware  reliability  standards  and  practice,  so  that  it 
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can  be  applied  in  a  unified,  integrated  approach  to  total  systems 
reliability.  To  the  end  of  moving  software  reliability 
engineering  towards  standardization  and  institutionalization,  a 
draft  military  handbook  on  combined  hardware/software  system 
reliability  prediction  and  estimation  is  being  written  as  part  of 
this  study  effort  (ELIN  A003) . 

1.3  Approach 

The  study  team  began  the  research  project  by  establishing  a 
software  error/ failure  database  for  use  in  developing  techniques 
for  system  decomposition  end  reliability  allocation,  reliability 
prediction  and  modeling,  reliability  growth  testing,  and 
reliability  demonstration.  The  team  populated  this  database  with 
error  data,  test  failure  data,  and  product/development  process 
metrics  from  eight  projects.  The  projects  were  selected  to 
encompass  a  wide  variety  of  size,  product,  and  development 
process  characteristics.  Two  of  the  projects  were  ongoing 
projects  whose  data  was  set  aside  to  be  used  solely  for  the 
validation  of  the  techniques. 

Since  the  database  required  data  from  several  life  cycle  phases 
of  each  project,  the  data  could  not  be  collected  all  at  once. 
The  data  collection  was  spread  out  over  several  months.  The 
prediction  task,  which  required  the  most  data,  was  the  last  task 
to  be  completed.  While  the  database  was  being  populated,  work 
began  on  the  allocation,  prediction,  growth,  and  demonstration 
tasks. 

System  decomposition  consists  of  allocating  functional 
requirements  to  hardware  and  software  subsystems  and  lower 
indenture  levels.  Reliability  allocation  consists  of 
apportioning  overall  system  reliability  requirements  to  those 
items.  For  software  the  indenture  levels  are  Computer  Software 
Configuration  Items  (CSCIs) ,  Computer  Software  Components  (CSCs) , 
Computer  Software  Units  (CSUs)  ,  and  modules.  The  approach  that 
was  taken  was  to  allocate  software  reliability  to  processes  (also 
called  tasks),  which  generally  correspond  to  the  CSCI  level. 

A  Markov  (combinations)  approach  for  repairable  systems  and  a 
combinatorial  approach  for  non-repairable  systems  were  used  for 
reliability  modeling.  Reliability  combination  (i.e,  calculation 
of  the  reliability  of  assemblages  of  hardware  and  software 
components)  was  studied  as  a  prerequisite  to  researching 
allocation. 

Reliability  prediction  was  approached  by  defining  the  problem  as 
predicting  the  parameters  of  a  suitable  software  reliability 
growth  model.  Predictive  relationships  based  on  regression 
analysis  of  the  product/process  metrics  and  error/failure  data 
were  developed. 
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For  growth  testing,  criteria  for  selecting  a  software  reliability 
growth  model,  goodness-of-f it  techniques,  and  recalibration 
techniques  were  developed.  Special  attention  was  paid  to 
staMstically  valid  ways  to  perform  the  testing,  including 
acceleration  by  means  of  multiple  copies. 

For  demonstration  testing,  an  appropriate  model  for  testing 
frozen  code  was  selected.  The  types  of  tests  developed  based  on 
that  model  were  fixed-length  tests,  failure-free  execution 
interval  tests,  and  sequential  tests.  Additional  types  of  tests 
were  developed  for  the  cases  of  demonstration  test  during  growth 
testing  and  for  using  multiple  copies. 

Each  of  the  techniques  developed  for  prediction,  decomposition 
and  allocation,  growth  testing,  and  demonstration  testing  were 
validated  on  ongoing  projects.  The  techniques  were  organized 
into  a  reliability  methodology  covering  the  entire  life  cycle. 
The  methodology  forms  the  basis  for  the  draft  MIL-HDBK  delivered 
under  contract  as  ELIN  A003. 


1.4  Report  Organization 

This  report  is  organized  to  provide  full  detailed  results  for  all 
items  in  the  SOW  of  contract  F30602-89-C-0111  which  are  not 
supplied  under  separate  CLIN  or  ELINs.  Coverage  of  the  study 
results  for  reliability  modeling  (SOW  Task  2) ,  allocation  (SOW 
Task  3),  prediction  (SOW  Task  4),  growth  testing  (SOW  Task  5), 
demonstration  testing  (SOW  Task  6),  and  validation  (SOW  Task  7) 
are  provided  in  the  main  body  of  the  report.  The  appendices 
provide  study  activity  reporting  for  reliability  data  base 
establishment  (SOW  Task  1)  and  provide  supplementary,  detailed, 
material  for  the  other  tasks  in  the  SOW  where  appropriate. 
Reporting  on  Reliability  Technique  Application  Software  (SOW  Task 
7)  and  Handbook  Guidelines  (SOW  Task  8)  is  not  provided.  SOW 
tasks  7  and  8  are  deliverable  under  CLIN  0001  and  CLIN  0002,  ELIN 
A003  respectively. 

1-5  Introductory  concepts 

1.5.1  software  Eailura 

Software  and  hardware  differ  in  several  respects.  Software  does 
not  wear  out;  almost  all  hardware  goes  through  a  wearout  phase. 
All  copies  of  software  are  perfectly  identical;  each  manufactured 
copy  of  a  piece  of  hardware  differs  to  some  extent.  Once  a  fault 
is  removed  from  software  it  is  gone  forever;  many  hardware  faults 
can  recur.  When  viewed  at  the  appropriate  level  of  abstraction, 
however,  hardware  and  software  reliability  are  very  similar. 
Both  a  running  program  and  an  operating  hardware  item  can  be  seen 
as  "black  boxes."  Every  once  in  a  while  the  black  box  fails. 
The  failure-inducing  stress  is  time.  For  software,  time  brings 
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with  it  a  succession  of  input  states.  The  more  time  that  goes 
by,  the  higher  the  quantity  of,  and  the  more  variety  of,  input 
states  the  program  encounters.  Eventually,  because  of  the 
presence  of  faults,  an  input  state  will  trigger  a  failure.  With 
hardware,  time  carries  with  it  random  stresses  (such  as  friction, 
shock,  corrosion)  which  gradually  or  suddenly  cause  failure. 
Thus  both  hardware  and  software  reliability  can  be  modeled  as 
random  or  stochastic  processes. 

An  interesting  trend  is  that  some  types  of  hardware  are  taking  on 
the  characteristics  of  software.  A  VLSI  chip  cannot  be 
exhaustively  tasted;  like  software  there  are  too  many  input 
states  and  too  many  paths.  Hermetically  sealed  integrated 
circuit  chips  are  claimed  to  have  no  wearout  failure  mode. 
Firmware  has  both  hardware  and  software  failure  modes.  While  a 
particular  version  of  a  program  does  not  wear  out,  program 
maintenance  is  an  entropy- increasing  process,  so  that  a  program 
will,  in  time,  deteriorate. 

A  software  failure  occurs  when  the  program  produces  output 
(display,  hardcopy,  command,  control,  etc.)  that  deviates  from 
what  the  requirements  specify.  A  failure  can  be  one  of 
conformance,  in  which  the  program  does  not  produce  the  right 
answer,  or  one  of  performance,  in  which  the  program  does  not 
perform  a  required  function  in  a  timely  or  resource-efficient 
manner.  Performance  failures  include  crashes,  hangs,  and 
software  that  does  not  meet  its  response  or  throughput  time 
requirements.  Real-time  systems  need  to  respond  to  events  in  the 
outside  world  as  those  events  are  happening. 

Software  failures  can  be  classified  along  several  different 
dimensions.  One  classification  is  by  the  number  of  discrepancies 
that  comprise  the  failure.  A  discrepancy  is  a  deviation  of  the 
value  of  a  single  output  variable  from  the  required  value. 
Another  way  failures  can  be  classified  is  by  the  severity  of 
their  consequences;  The  impact  of  a  failure  could  be  as 
innocuous  as  a  misspelling  or  as  catastrophic  as  loss  of  life  or 
limb. 

Software  reliability  is  one  measure  of  software  quality. 
"Software  quality"  refers  to  those  attributes  that  a  program  is 
required  to  possess  at  all  times.  Examples  include  security, 
robustness,  maintainability,  safety,  availability,  and 
portability.  Quantitatively,  software  reliability  is  defined  as 
the  probability  of  failure-free  operation  of  the  software  for  a 
specified  period  of  time,  in  a  specified  environment.  An 
alternative  figure  of  merit  is  the  software's  failure  rate,  the 
instantaneous  rate  of  software  failures  per  unit  time.  By  "time" 
is  meant  execution  time.  The  failure-inducing  stress  is 
execution  time,  since  software  that  is  not  running  cannot  fail. 
When  combining  software  failure  rates  with  one  another  and  with 
hardware,  the  failure  rate  is  re-expressed  with  respect  to  system 
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operating  time.  The  level  at  which  software  reliability  should 
be  expressed  needs  to  be  a  high  one,  such  as  Computer  Software 
Configuration  Item  (CSCI) ,  so  that  the  interfaces  between  lower 
level  items  will  be  included. 

Software  failures  arise  from  a  population  of  software  faults.  A 
software  fault  (often  called  ''bug")  is  missing,  extra,  or 
defective  code  that  has  caused  or  can  potentially  cause  a 
failure.  Every  time  a  fault  is  traversed  during  execution,  a 
failure  does  not  necessarily  ensue;  it  depends  on  the  machine 
stare  (values  of  intermediate  variables) .  The  extent  to  which  a 
source  program  contains  faults  is  indicated  by  its  fault  density, 
expressed  in  faults  per  thousand  lines  of  executable  3ource  code 
(KLOC) .  The  fault  density  does  not  translate  directly  into  a 
failure  rate  because  different  parts  of  the  program  are  executed 
with  different  frequencies.  Instructions  inside  loops  will  tend 
to  be  executed  more  often  than  instructions  outside  loops. 
Conditionally  executed  instructions  tend  to  get  executed  less 
than  unconditional  ones.  How  many  loop  iterations  occur  and 
which  branches  take  place  depend  on  the  machine  state,  which  in 
turn  depends  on  the  input  state. 

The  origin  of  software  faults  lies  in  human  fallibility. 
Compounding  factors  are  the  lagging  of  software  engineering 
technology  behind  the  increasing  capabilities  of  computers,  the 
lr.rge  number  of  discrete  states  in  programs,  the  novelty  of 
problems  and  solutions,  and  lack  of  standard  off-the-shelf 
software  "parts"  and  designs.  Software  is  very  flexible  because 
there  are  no  physical  constraints  such  as  power,  weight,  quantity 
of  parts,  manufacturability,  etc.,  so  software  engineers  take  on 
tasks  of  sometimes  enormous  size  and  complexity. 

1-5.2  sattms  Design  ,.,APRr<?aghga.T9  Reliability 

High  levels  of  software  reliability  are  accomplished  through  a 
triad  of  activities:  fault  avoidance,  fault  elimination,  and 
fault  tolerance. 

Fault  avoidance  consists  of  applying  sound  software  engineering 
practices,  including  standards  (documentation,  structured  design 
and  programming,  control),  quality  assurance  (formal  reviews, 
audits;  evaluation  of  personnel,  methods),  and  verification  a.;d 
validation.  These  methods  can  go  a  long  way  in  keeping  down  the 
number  of  faults  but,  historically,  many  faults  will  remain. 

Fault  elimination  is  accomplished  through  testing,  code  reading, 
and  walkthroughs.  The  only  way  to  uncover  and  remove  all  faults 
in  the  code  is  through  exhaustive  testing.  Exhaustive  testing  of 
non-trivial  programs  is  impossible  from  a  practical  point  of  view 
because  the  number  of  inputs  states  is  astronomically  large. 
Formal  proof  of  correctness,  given  the  current  state  of  the  art, 
is  likewise  impractical  for  real-world  software.  In  a  program 
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correctness  proof,  the  program  is  treated  as  a  static 
mathematical  object.  The  correctness  proof  is  a  formal 
mathematical  demonstration  that  a  program  is  consistent  with  its 
specification.  Such  proofs  are  of  great  size  and  complexity , 
although  mechanical  verifier  systems  help.  A  formal 
specification  of  the  program  is  required,  and  it  is  impossible  to 
demonstrate  whether  the  specification  captures  the  intentions  of 
the  customer.  Even  if  exhaustive  testing  or  a  formal  proof  of 
correctness  could  be  accomplished,  the  results  of  the  activity 
are  as  vulnerable  to  hvuaan  fallibility  as  programming  is. 

Fault  tolerance  is  achieved  through  special  programming  tech¬ 
niques  that  enable  the  software  to  detect  and  recover  from 
failure  incidents.  Software  fault  tolerance  is  a  controversial 
topic.  The  method  requires  "redundant"  software  elements  that 
provide  alternative  means  of  fulfilling  the  same  function.  The 
different  versions  must  be  such  that  they  will  not  all  fail  in 
response  to  the  same  circumstances.  Some  have  suggested  that 
diverse  software  versions  developed  using  different  specifica¬ 
tions,  designs,  programming  teams,  programming  languages,  etc., 
might  fail  in  a  statistically  independent  manner.  Empirical 
evidence  questions  that  hypothesis.  However,  almost  all  software 
fault-tolerance  experiments  have  reported  some  degree  of 
reliability  improvement.  Despite  advances  in  software  fault 
avoidance,  elimination,  and  tolerance,  large-scale  software  will 
have  faults  and  it  will  occasionally  fail.  There  is  a  need  to 
specify,  predict,  measure,  allocate,  model,  and  demonstrate 
software  reliability.  Software  reliability  figures  need  to  be 
meaningfully  combined  with  hardware  reliability  figures  to  yield 
system  reliability  figures. 

1.5.3  Software  Reliability  Growth 

For  the  results  of  growth  model  parameter  estimation  and  software 
reliability  growth  testing  to  be  valid,  it  is  important  that  the 
environment  during  test  be  the  same  as  the  actual  field  use 
environment.  The  hardware  platform  (actual  or  emulated)  and  the 
system  software  (such  as  the  operating  system  version)  must  be 
the  same.  A  very  important  part  of  the  environment  is  the 
program's  operational  profile.  An  operational  profile  associates 
each  point  (input  state)  in  the  program's  input  space  with  a 
probability  of  occurrence.  The  operational  profiles  during 
testing  and  field  use  must  be  identical.  Since  the  input  space 
of  most  programs  is  quite  large,  fully  specifying  the  operational 
profile  is  impractical.  An  operational  profile  can  usually  be 
expressed  as  the  relative  frequencies  of  end-user  functions, 
resulting  in  a  "functional  profile."  For  real-time  systems, 
ordering  and  timing  of  inputs  may  also  enter  into  a  description 
of  the  environment. 

Just  because  faults  lurk  in  the  code,  however,  does  not  mean  that 
the  program  will  provide  unreliable  service  to  end-users.  Each 
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user' 8  individual  operational  profile  will  result  in  a  different 
rate  at  which  faults  are  encountered.  The  operational  profile 
determines  the  probability  of  each  input  state  and  hence  each 
path  through  the  program.  The  input  state  determines  the  machine 
state  (values  of  all  intermediate  variables)  the  computer  will  be 
in  when  the  fault  is  encountered.  In  certain  machine  states  the 
fault  will  cause  a  failure. 

During  a  hardware  item's  useful  life  (between  bum-in  and 
wearout)  a  constant  failure  rate  model  is  generally  employed. 
For  a  program  whose  code  is  frozen,  subjected  to  input  randomly 
selected  from  a  fixed  operational  profile,  a  constant  failure 
rate  model  is  a  reasonable  one  (see  Appendix  B) .  During  system 
test,  as  the  code  is  altered  as  the  result  of  fault  correction 
activity,  the  failure  rate  will  vary  as  a  function  of  cumulative 
execution  time. 

The  failure  rate  of  a  piece  of  software  is  a  function  of  the 
number  and  location  of  faults  in  the  code,  how  fast  the  program 
is  being  executed,  and  the  operational  profile.  Faults  cannot 
be  directly  observed,  if  only  for  the  reason  that  any  faults 
whose  whereabouts  are  known  would  presumably  have  already  been 
removed.  The  way  to  find  faults  is  to  execute  the  software 
starting  from  various  input  states.  When  a  failure  occurs  the 
symptoms  are  recorded.  Fault-correction  personnel  analyze  the 
symptoms  and  look  at  the  code  to  try  and  locate  the  fault  that 
caused  the  failure.  If  the  fault  is  not  obvious,  the  personnel 
will  try  to  reproduce  the  failure,  this  time  running  with  a 
debugging  tool  or  having  inserted  additional  output  statements 
into  the  program.  A  debugging  tool  typically  allows  the  user  to 
execute  the  program  a  step  at  the  time  and  lets  him  or  her 
examine  and  deposit  variable  values.  Usually,  the  programmer 
finds  the  fault  and  removes  it.  At  times  the  repair  activity 
introduces  new  faults  into  the  code. 

While  repair  activity  is  imperfect,  the  hoped-for  and  generally 
observed  result  is  that  the  times  between  failures  tend  to  grow 
longer  and  longer  as  the  process  of  testing  and  fault  correction 
goes  on.  A  software  reliability  growth  model  mathematically 
summarizes  a  set  of  assumptions  about  the  phenomenon  of  software 
failure.  The  model  provides  a  genera]  form  for  the  failure  rate 
as  a  function  of  time  and  contains  unknown  parameters  that  are 
determined  either  by  prediction  or  estimation. 

Growth  testing  and  modeling  is  used  to  estimate  the  time  and 
effort  needed  to  reach  intermediate  or  required  reliability  goals 
and  to  track  rhe  progress  the  software  is  making  toward  those 
reliability  goals.  If  the  software  is  falling  short,  management 
can  re-allocate  resources  or  take  other  corrective  action. 
Growth  testing  thus  provides  visibility  to  management  about  where 
the  software  reliability  is  currently  and  where  it  is  expected  to 
be  at  a  given  milestone. 
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A  software  reliability  growth  model  makes  assumptions  about  the 
nature  of  the  distribution  that  fits  the  failure  data.  Since 
there  is  usually  no  prior  statistical  information  about  the 
failure  behavior  of  a  specific  program,  a  goodness-of-fit 
procedure  can  be  applied  to  assess  how  well  the  model  is  fitting 
the  data.  If  the  fit  is  not  good,  an  alternate  model  or 
statistical  estimation  technique  can  be  employed  or  a 
recalibration  technique  can  be  applied  to  the  model  to  improve 
the  results. 

1.5.4  Basic  Reliability  Concepts 

Quantitatively,  the  reliability  of  an  item  is  expressed  by  its 
reliability  function  R(t) .  To  be  meaningful,  the  environment  in 
which  the  item  operates  must  be  specified.  The  reliability 
function  gives  the  probability  that  the  item  will  fail  by  time  t. 
The  reliability  of  an  item  is  usually  specified  for  mission 
oriented  systems.  An  alternative  way  of  expressing  an  item's 
relative  reliability  is  by  its  failure  rate  X(t),  which  is  the 
rate  of  failures  per  unit  time  at  the  instant  t.  In  situations 
where  the  failure  rate  is  a  constant,  1,  the  reliability  function 
and  failure  rate  are  related  by  the  equation  R(t)  -  exp[-lt] . 
MTBF,  the  reciprocal  of  failure  rate  or  availability  are  the  most 
common  reliability  figures  of  merit  for  systems  which  are 
continuously  operated  and  undergo  repair  when  failures  occur. 


8 


2.  SYSTEM  AND  SOFTWARE  MODELING/COMBINATION 


Reliability  modeling  of  combined  hardware  and  software  systems  is 
analogous  to  reliability  modeling  of  purely  hardware  systems. 
The  failure  rates  for  both  hardware  and  software  are  treated  as 
constant  (see  Appendix  B) .  Reliability  block  diagrams  of  system 
elements  are  developed  and  employed.  Individual  hardware 
platforms  and  the  software  assigned  to  those  platforms  are 
independent  of  other  hardware/software  platforms.  State  diagrams 
that  accurately  portray  the  interrelationship  between  the 
hardware  platforms  and  the  software  executing  on  the  platforms 
are  developed  and  used  in  estimating  reliability  figures  of 
merit . 

This  section  provides  a  summary  overview  of  the  techniques 
applicable  to  the  reliability  modeling  of  combined  hardware  and 
software  systems.  An  abbreviated  overview  of  the.  system  modeling 
process  is  provided.  This  overview  is  used  to  help  the  analyst 
identify  those  system  properties  which  are  unique  to  combined 
hardware  and  software  systems.  The  majority  of  this  section  is 
dedicated  to  describing  the  development  of  software  failure  rates 
that  are  a  composite  of  the  multiple  processes  that  may  be 
executing  during  any  time  period.  A  major  treatment  of 
reliability  modeling  of  HW/SW  systems  was  provided  in  [James  et 
al .  (1982)].  This  report  provides  modest  clarification  to  the 
major  work  on  reliability  modeling  that  has  been  previously  done. 
Specifically,  some  discussion  of  the  reliability  modeling  detail 
as  it  relates  to  active  redundancy  is  provided. 

Modeling  methods  used  to  model  combined  HW/SW  systems  for  the 
purposes  of  reliability  estimation  and  allocation  need  to 
accurately  assess  the  interdependence  between  individual  software 
elements,  the  hardware  platforms  on  which  these  software  elemonts 
execute,  and  the  services  provided  by  the  system  being  analyzed. 
Additionally,  the  methods  used  need  to  be  based  on  and  compatible 
with  modem  system  engineering  methods. 

Reliability  modeling  is  based  on  system  FMEA  (Failure  Modes  and 
Effects  Analysis),  a  traditional,  bottom  up,  reliability  analysis 
technique  that  provides  a  mapping  between  failures  and  their 
impact  on  system  services.  These  FMEAs  need  not  always  be  a 
formal  analysis  since  the  results  are  expressed  in  the 
reliability  model  block  diagrams.  Both  fault  tree  analysis  and 
FMEA  were  critically  examined  for  use  in  combined  HW/SW  system 
reliability  model  development.  Either  fault  tree  analysis  or 
FMEA  can  produce  the  needed  analysis  of  the  dependencies  between 
system  hardware  elements  and  system  services.  However,  fault 
tree  analysis  of  the  loss  of  each  system  service,  while  accurate 
and  complete,  creates  a  large  amount  of  duplication  of 
information.  FMEA  techniques  tend  to  result  in  a  more  compact 
display  of  the  needed  data.  However,  traditional  FMEAs  that  are 
entered  on  multiple  pages  of  FMEA  forms  create  a  large  amount  of 
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data  that  is  poorly  organized  for  reliability  model  development. 
The  use  of  a  system  level  adaptation  of  Matrix  FMEA  techniques 
[Barbour  (1977),  Goddard  (1984)]  results  in  a  compact  readily 
usable  display  of  the  needed  FMEA  information. 

The  modeling  of  combined  HW/SW  systems,  whether  for  reliability 
allocation  or  estimating  purposes,  is  best  approached  on  a 
functional  service  basis  using  a  matrix  FMEA  approach.  The 
resultant  FMEA  can  then  be  used  to  develop  a  HW/SW  system 
reliability  block  diagram  of  independent  elements.  The 
individual  series/parallel  elements  of  the  reliability  block 
diagram  can  then  be  modeled.  Non-redundant  systems  can  be 
modeled  as  series  strings  of  hardware  and  HW/SW  system  elements. 
Complex,  redundant  systems  and  system  elements  are  modeled  using 
Markov  state  diagrams  to  accurately  portray  the  possible 
operational  and  non-operational  states.  In  general,  these  state 
diagrams  will  be  complex  enough  to  require  access  to  automated 
tools  for  solution.  They  are  not,  strictly  speaking, 
intractable.  However,  the  labor  required  to  manually  determine  a 
specific  closed  form  solution  for  a  state  diagram  that  has  been 
developed  to  model  a  specific  design  being  analyzed  is  usually 
prohibitive.  Automated  solutions  of  these  state  diagrams  are 
possible  both  analytically  and  through  simulation.  Tools  for 
analytic  and  simulation  solutions  of  Markov  state  diagrams  are 
available  to  government  offices  and  their  contractors  through 
Rome  Labs  and  NASA  at  nominal  costs.  The  user  of  these  tools 
will  need  to  determine  whether  or  not  the  numerical  accuracy 
needed  for  their  specific  situation  is  supported.  Analytic 
solutions  to  these  diagrams  often  require  the  solution  of  a 
transition  matrix  with  potentially  significant  losses  in 
numerical  accuracy  due  to  the  multiple  arithmetic  operations 
compounded  by  the  accuracy  limitations  of  the  processing  platform 
being  used.  Monte  Carlo  solutions  to  Markov  chains  can  result  in 
both  numerical  inaccuracy  due  to  the  methods  employed  and  may 
involve  substantial  cost  for  multiple  program  runs.  The  costs 
associated  with  Monte  Carlo  solutions  rise  dramatically  as  the 
degree  of  accuracy  demanded  increases. 

Estimation  of  system  reliability  characteristics  is  based  on  the 
reliability  block  diagrams  and  Markov  state  diagrams  developed 
from  the  system  services  FMEA  and  the  individual  hardware  and 
software  component  reliability.  Estimation  techniques  for 
hardware  reliability  and  maintainability  characteristics  are  well 
known  and  can  be  applied  to  the  hardware  portions  of  combined 
HW/SW  systems.  Software  reliability  characteristics  can  be 
estimated  using  the  methods  described  in  this  report.  For 
redundant,  fault  tolerant  systems,  software  recovery 
characteristics  are  system  design  and  implementation  dependent. 
These  recovery  characteristics  will  need  to  be  estimated  on  a 
case  by  'case  basis  in  conjunction  with  performance  modeling  and 
estimation. 
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Software  maintainability,  the  time  required  to  isolate  and 
correct  a  fault  in  the  design,  is  not  used  in  the  reliability 
modeling  and  allocation  discussed  in  this  section.  Software 
maintenance  is  expected  to  proceed  in  parallel  with  ongoing 
system  operation  following  a  software  failure.  Thus,  the  time 
required  to  reestablish  system  operation  following  a  software 
failure  is  used  as  the  repair  or  recovery  rate  in  the  modeling  of 
software  elements  of  combined  HW/SW  elements.  Software 
maintenance  will  result  in  a  software  failure  rate  that  is  not 
constant  over  time  due  to  the  software  corrections  being 
implemented.  However,  for  the  purposes  of  modeling  and 
allocation  of  combined  hardware  and  software  systems,  an 
assumption  of  constant  software  failure  rata  at  any  operational 
period  (i.e.,  between  software  fixes)  is  justified.  Estimations 
of  the  software  failure  rate  and  of  software  reliability  growth 
rates  are  discussed  elsewhere  within  this  report. 

2.1  Development  of  the  System  Model 

The  development  of  an  accurate  and  representative  system 
reliability  model  for  use  in  reliability  estimation  and 
allocation  is  dependent  on  a  thorough  understanding  of  the  system 
being  modeled.  First  a  FMEA  is  developed  for  the  system  design 
which  relates  the  services  provided  by  the  system  to  the  various 
hardware  and  software  components.  This  FMEA  is  then  translated 
into  a  system  model. 

2.1.1  Developing  the,, System JEMEA 

Development  of  a  reliability  model  for  use  in  reliability 
estimation  and  allocation  begins  with  the  use  of  the  functional 
decomposition  that  has  been  developed  as  a  part  of  the  system 
engineering  process.  For  small  or  relatively  simple  system 
structures,  system  functional  analysis  may  have  been  omitted  as  a 
formal  procedure.  If  the  system  level  functional  decomposition 
is  not  available,  the  reliability  engineer  may  find  it  necessary 
to  recreate  this  analysis  using  either  Data  Flow  Diagrams 
[DeMarco  (1978)]  or  using  the  Real  Time  System  Specification 
Strategies  of  Hatley  and  Pirbhai  [Hatley  (1987)].  The  functional 
decomposition  of  the  system  is  used  to  identify  the  hardware 
configuration  items  (HWCIs) ,  the  computer  software  configuration 
items  (CSCIs)  along  with  the  processing  provided  by  these  CSCIs, 
and  the  allocation  of  CSCIs  to  various  HWCIs  within  the  system. 
The  analyst  can  then  begin  to  create  the  system  level  FMEA  that 
will  support  reliability  modeling  of  the  combined  HW  and  SW 
system. 

The  system  level  FMEA,  shown  diagrammatical ly  in  Figure  2-1,  is  a 
mapping  of  the  hardware  and  software  components  of  the  system 
onto  the  system  services  provided.  To  create  the  FMEA,  the 
analyst  first  constructs  a  matrix  with  each  of  the  hardware  CIs 
and  their  associated  software  CSCIs  or  CSCs  as  appropriate  along 
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the  vertical  axis.  The  horizontal  axis  is  formed  by  the  system 
services  or  outputs  of  the  system  grouped  in  convenient  ways  that 
support  the  desired  analysis.  A  grouping  of  system  services  by 
system  operating  mode  often  supports  development  of  the  various 
models  required  by  the  system  specification.  The  HWCIs  and  CSCls 
(CSCs)  are  then  mapped  onto  the  system  services  or  outputs  based 
on  the  impact  on  system  services  caused  by  the  failure  of  each 
hardware  and  software  element.  In  performing  this  mapping,  the 
analyst  will  need  to  assess  the  impact  of  failure  of  both 
hardware  platforms  and  software  elements.  The  failure  impact  of 
software  elements  will  need  to  be  examined  in  depth  based  on  the 
data  flow  that  has  been  established  for  the  system  design. 
Similarly,  the  failure  of  hardware  platforms  will  need  to  be 
examined  for  its  impact  on  software-provided  services  using  the 
data  flow  diagrams  for  the  system  software  resident  on  the 
hardware  platform. 

Once  the  system  level  FMEA  has  been  completed,  the  analyst  can 
examine  the  hardware  and  software  that  is  required  for  any 
particular  mode  or  set  of  system  services.  A  reliability  model 
for  these  services  can  then  be  developed. 

2.1.2  sm,6m Jjadfll ,  Bmloanann 

2.1.2. 1  Reliability  Block  Diagram  Development 

The  reliability  block  diagram  for  combined  HW/SW  systems  is 
developed  based  on  the  system  FMEA,  using  a  procedure  that  is 
analogous  to  that  used  for  purely  hardware  systems.  The  FMEA 
results  are  used  to  determine  which  hardware  and  software 
elements  are  required  to  provide  a  set  of  system  services  of 
interest.  The  analyst  then  proceeds  to  develop  a  reliability 
block  diagram  that  consists  of  a  set  of  series  blocks  for  each  of 
the  independent  HW/SW  subsystems  or  elements  that  must  be 
operational  to  provide  the  services  being  modeled.  In  general, 
the  analyst  should  only  separate  software  elements  from  the 
hardware  elements  on  which  they  execute  as  a  final  step  in  the 
reliability  block  diagram  modeling.  Distinct  separation  of 
hardware  platforms  and  the  software  that  executes  on  those 
platforms  is  based  on  system  application  specific  information 
that  is  used  in  developing  a  state  diagram (s)  that  accurately 
represents  the  interaction  between  the  hardware  platform  and  the 
software  executing  on  it. 
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2. i.2. 2  stafcfl  Diagram,. BfvtiflBminfc 

The  development  of  one  or  more  state  diagrams  that  accurately 
represent  the  hardware  and  software  interactions  for  all  system 
processing  elements,  both  series  and  redundant,  is  the  next  step 
in  developing  reliability  models  of  combined  HW/SW  systems.  The 
notation  that  will  be  used  for  the  presentation  and  discussion  of 
state  diagrams  in  this  section  of  the  report  is  given  in  Figure 
2*2.  The  reader  should  review  that  figure  for  notation  prior  to 
proceeding  with  the  remainder  of  this  section. 
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STATEN:  |N| 

STATE  TRANSITION  MTO  N  :  (M,  N) 


I 


DEFINITIONS 


1  HW  -  HARDWARE  FAILURE  RATE  |i  HW  ■  HARDWARE  REPAIR  RATE 

\  8W  •  SOFTWARE  FAILURE  RATE  H  SW  -  SOFTWARE  REPAIR  RATE 

TR  •  TIME  REQUIRED  FOR  MANUAL  RECOVERV  Cd-  FAULT  DETECTION  COVERAQE 

C  |  -  FAULT  ISOLATION  COVERAQE 
C  r  •  FAULT  RECOVERY  COVERAQE 


Figure  2-2.  Notation  Used  in  State  Diagrams  and  Text 

Software  restart,  which  is  used  in  the  state  diagrams  discussed 
below,  ia  tha  time  raquirad  to  raatart  tha  faiiad  aoftwara 
without  changing  tha  aoftwara  itaalf.  As  diacuaaad  previously, 
aoftwara  rapair  rataa  ara  not  uaad  in  tha  atata  diagrams 
diacuaaad  in  this  aaction.  Instead,  tha  aoftwara  racovary  rata 
ia  adjusted  to  include  those  caaaa  where  aoftwara  must  ba 
"patched"  prior  to  successful  restart.  Software  failures  ara 
assumed  to  ba  dependant  on  the  state  of  the  software  execution 
environment  at  the  time  of  failure.  Thus,  the  failed  software 
can  often  ba  restarted,  without  any  change  to  the  software 
itself,  and  will  reaume  operation  as  long  as  tha  environment  that 
was  in  existence  at  the  time  of  tha  aoftwara  failure  has  changed 
sufficiently.  Successful  restart  after  failure  is  a  reasonable 
assumption  for  software  with  moderate  maturity.  The  software 
that  has  faiiad  may  have  operated  successfully  for  tens  to 
hundreds  of  hours  prior  to  being  presented  with  a  specific 
processing  system  state  and  set  of  inputs  that  ware  sufficient  to 
cause  the  failure  to  be  manifested.  However,  successful  restart 
of  tha  failed  software  will  depend  on  the  persistence  of  the  set 
of  execution  conditions  that  caused  the  original  failure.  If  tha 
persistence  of  the  failure  is  sufficiently  long,  repair  of  the 
software  may  be  required  prior  to  any  restart.  For  the  purpose 
of  reliability  modeling  of  HW/SW  systems,  a  long  persistence 
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failure  is  one  that  will  continue  to  cause  software  failure  when 
restart  is  attempted.  Any  set  of  execution  conditions  that 
persists  long  enough  to  cause  a  failure  upon  attempted  restart  is 
expected  to  result  in  system  shutdown  for  most  non-redundant 
systems  and  effective  loss  of  redundancy  in  cases  of  software 
failure  for  redundant  systems.  The  relative  fractions  of  long 
persistence  to  short  persistence  failures  for  mature  software  are 
not  known.  James  et  al.  (1982)  reported  an  average  of  GO  percent 
short  persistence  failures  for  immature  software  undergoing 
factory  test. 

The  time  required  to  restart  software  with  a  short  persistence 
failure  may  vary  widely  and  may  exceed  the  repair  time  required 
for  hardware  failures  in  some  cases.  Long  restart  times  are 
expected  to  occur  for  failures  that  "crash"  the  operating  system 
leaving  system  resources  in  an  indeterminate  state  that  must  be 
determined  and  corrected  (if  needed)  manually.  Software  failures 
that  leave  large  data  stores  inconsistent  are  also  expected  to 
require  long  restart  times  due  to  the  difficulty  in  restoring  the 
data  to  a  known  acceptable  state.  For  example ,  failures  that 
cause  a  data  base  to  lose  integrity  may  require  restoration  from 
a  backup  tape  with  subsequent  update  using  journal  files. 
However ,  the  average  time  required  for  software  restart  is 
expected  to  be  less  than  the  average  hardware  repair  time  since 
restart  of  all  system  software  is  included  in  the  hardware 
maintenance  time  and  at  least  some  hardware  failures  may  leave 
system  resources  and/or  data  stores  in  an  inconsistent  state. 

2. 1.2. 2.1  series,  ..ELamflirt  a 

The  state  diagrams  used  to  represent  series  HW/SW  elements  are 
simple  state  models  as  shown  in  Figure  2-3.  The  hardware  and 
software  elements  can  be  treated  as  independent  paths  that  can 
lead  to  failure  since  both  hardware  and  software  must  be  fully 
operational  for  the  system  to  be  in  a  success  state.  The  three 
possible  state  models  are  shown  in  the  figure  as  (a)  for  systems 
undergoing  a  continuous  operation  and  repair  cycle,  (b)  non¬ 
repayable  systems  that  are  not  self  healing  in  the  case  of 
software  failure  (i.e.,  single  shot  systems),  and  (c)  non¬ 
repayable  systems  with  self-healing  for  software  failures. 

As  shown  in  part  "a"  of  the  figure,  repairable  systems  experience 
hardware  and  software  failures  and  recover  independently  through 
either  hardware  repair  or  software  restart.  The  recovery  time 
for  software  in  this  model  includes  the  impact  of  any  long 
persistence  software  failures  (i.e.,  those  which  cannot  be 
repaired  through  restart) .  At  any  time  that  either  the  hardware 
or  software  has  failed  and  not  yet  been  repaired  or  restarted, 
the  system  is  considered  to  be  in  a  failed  state.  It  is  possible 
to  transition  from  state  2  (software  failure)  to  state  1 
(hardware  failure)  if  a  hardware  failure  occurs  prior  to 
successful  software  restart.  Since  the  hardware  remains 
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operational  after  a  software  failure ,  there  is  a  continuous 
exposure  to  the  possibility  of  hardware  failure.  The  transition 
(2,1)  can  usually  be  safely  ignored  without  undue  loss  of 
accuracy  since  (2,1)<<(2,0)  in  most  cases.  The  reliability 
metrics  of  interest  for  repairable  systems  (e.g.,  availability, 
MTBF)  can  be  derived  from  the  closed  form  semi-Markov  process 
shown. 


(a)  REPAIRABLE  SYSTEM  (b)  NON-REPAIRABLE 

SYSTEM 


(o)  NON-REPAIRABLE 
SY3TEM  WITH  SW 
RESTART 


Figure  2-3.  State  Diagrams  for  Simple  Series  HW/SW  Elements 
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Series  non-repairable  system  elements  are  modeled  as  shown  in 
part  "b"  of  the  figure.  The  system  element  is  considered  fully 
operational  as  long  as  both  the  hardware  and  software  remain 
operational.  If  a  failure  of  either  hardware  or  software  occurs, 
the  system  transitions  from  state  0  to  state  1,  which  is  an 
absorbing  failed  state.  The  state  diagram  shown  allows  the 
calculation  of  relevant  reliability  metrics  of  interest  (MTTF, 
Reliability,  Probability  of  Success)  through  evaluation  of  the 
probability  of  the  transition  (0,1)  occurring  during  any  given 
time  period  tl  to  t2.  In  practice,  completely  non-repairable 
systems  are  relatively  rare.  This  type  of  system  would  be 
expected  to  be  found  in  systems  where  the  critical  duration  of 
operation  is  too  short  to  allow  software  restart  to  be  attempted. 
Most  critical  real  time  military  systems  of  interest  are  expected 
to  include  the  ability  to  restart  software.  ^ 

Series  non-repairable  elements  with  software  restart  are  modeled 
using  the  state  diagram  shown  in  part  "c"  of  the  figure. 
Transitions  from  successful  operation  (state  0)  to  either 
hardware  failure  (state  1)  or  software  failure  (state  2)  occur  at 
the  respective  failure  rates  shown.  Hardware  failures  cause  the 
system  to  enter  a  permanent  failed  state  (state  1)  that  is 
absorbing.  Software  failures  cause  the  system  to  enter  a 
software  failed  state  (state  2)  .  The  element  then  restarts  the 
software  in  an  attempt  to  resume  operation.  This  type  of  system 
operation  is  commonly  found  in  aircraft  avionics  that  have 
software  (firmware)  embedded  into  the  equipment.  Typically,  upon 
failure  of  a  hardware  or  software  element,  the  equipment  removes 
itself  from  any  active  operation,  flags  the  pilot  that  an  error 
has  occurred,  and  enters  a  self  test  mode.  Following  successful 
completion  of  the  self  test,  a  restart  is  attempted.  If  the 
failure  was  induced  into  hardware  by  the  environment  or  was  a 
short  persistence  software  failure,  an  equipment  restart  will  be 
successful,  allowing  the  equipment  to  resume  operation. 

2. 1.2. 2. 2  Redundant Jilmanfcft 

Reliability  models  of  redundant  HW/SW  elements  are  significantly 
more  complex  than  reliability  models  of  series  elements.  The 
addition  of  redundancy  introduces  complexity  associated  with  the 
ability  of  the  HW  and  SW  to  correctly  respond  to  failure  events. 
Reliability  modeling  of  redundant  HW/SW  elements  with  hot  standby 
and  automatic  switchover  capability  significantly  increases  the 
number  of  states  required  to  properly  account  for  system 
behavior. 

2. 1.2. 2. 2.1  Hardware  Systems 

A  general  model  for  hardware  redundancy  using  identical  equipment 
is  shown  in  Figure  2-4.  As  shown  in  the  figure,  redundant  system 
elements  transition  to  the  next  higher  state  upon  the  occurrence 
of  any  HW  failure.  Hardware  repairs  transition  the  system 
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element  model  to  the  next  lower  (numerically)  state.  The  system 
is  a  closed  form  semi-Markov  process  that  can  be  solved  for  the 
appropriate  reliability  measures  using  conventional  methods. 
Closed  form  solutions  for  the  reliability  measures  of  interest 
for  this  type  of  model  under  most  common  repair  restrictions, 
types  of  standby,  etc,  are  available  in  the  literature  [Kozlov 
and  Ushakov  (1970)]. 

The  model  shown  in  Figure  2-4  provides  an  upper  bound  on  the 
reliability  of  redundant  hardware  systems.  Estimation  of  the 
expected  reliability  of  hardware  systems  requires  that  the  fault 
tolerance  employed  in  the  redundancy  be  included  in  the  model. 
For  cold  standby  system,  where  backup  elements  are  not  powered 
and  thus  immune  to  failure  occurrence,  the  model  of  Figure  2-4 
provides  a  reasonable  estimate  if  the  transition  rates  shown  from 
each  success  state  to  the  next  higher  number  state  are  adjusted 
to  account  for  the  constant  number  of  elements  in  operation  (m 
units) .  However,  for  hot  standby  systems  with  automatic 
switchover,  the  model  of  Figure  2-4  significantly  overstates  the 
reliability  achieved  by  the  redundant  hardware  elements. 
Failures  in  the  fault  detection  mechanisms  that  may  lead  to 
latent  faults  in  backup  equipment,  as  well  as  failures  in  fault 
detection,  fault  isolation,  and  fault  recovery  mechanisms  that 
may  lead  to  an  inability  to  activate  redundant  system  elements 
and  resume  system  services  in  response  to  primary  element 
failures,  are  not  included  in  the  model  shown  in  the  figure. 

Figure  2-5  is  a  simplified  reliability  model  for  a  hardware 
system  employing  hot  standby,  and  automatic  switchover  with  one 
of  two  identical  elements  required.  The  model  accounts  for 
failures  in  the  fault  detection,  isolation,  and  recovery 
mechanisms.  The  concept  of  three  types  of  "coverage"  is 
introduced  as  a  part  of  the  model.  Fault  detection  coverage  (Cd) 
is  the  probability  of  detecting  a  fault  given  that  a  fault  has 
occurred.  Fault  Isolation  coverage  (Ci)  is  the  probability  that 
a  fault  will  be  correctly  isolated  to  the  recoverable  interface 
(level  at  which  redundancy  is  available)  given  that  a  fault  has 
occurred  and  been  detected.  Fault  recovery  coverage  is  the 
probability  that  the  redundant  structure  will  recover  system 
services  given  that  a  fault  has  occurred,  been  detected,  and 
correctly  isolated.  The  model  shown  in  Figure  2-5  is  a 
simplified  model  since  it  does  not  separately  consider  the 
possible  impact  of  transient  failures.  The  model  also  assumes 
that  fault  detection  coverage  (Cd)  is  the  same  for  both  the 
primary  element  and  the  backup  element.  In  practice,  there  may 
be  different  levels  of  fault  detection  coverage  between  primary 
and  backup  equipments  due  to  a  difference  in  test  exposure 
intensity. 
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N*hw 


Figure  2-4.  General  Hardware  Redundancy  Model:  M  Required  of  N  Supplied  Identical  Elemen 


Figure  2-5.  Hardware  Reliability  Model  for  1:2  Identical  Elements  Hot  Standby  with 
Automatic  Switchover 
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As  shown  in  the  reliability  model,  the  structure  can  transition 
from  the  full  up  state  (1)  to  one  of  three  states.  The  structure 
transitions  to  state  1  whenever  a  hardware  failure  occurs  in  the 
primary  element  that  is  correctly  detected,  isolated,  and 
recovered  from.  Similarly,  a  detected  failure  in  the  backup 
element  results  in  a  transition  from  state  0  to  state  1.  The 
structure  transitions  from  state  0  to  state  4  when  a  failure 
occurs  in  the  backup  equipment  which  is  not  detectable.  Failures 
in  the  primary  hardware  element  that  cannot  be  correctly 
detected,  isolated,  or  recovered  from  result  in  a  transition  from 
state  0  to  state  3.  State  3  is  a  system  state  to  account  for  the 
failure  time  accrued  during  manual  intervention  by  the  system 
operator  to  restore  lost  system  services.  Transitions  from 
states  1,  3,  and  4  to  state  2  are  caused  by  a  hardware  failure 
occurring  prior  to  repair  of  the  first  failure  which  occurred  in 
the  system  structure. 

In  actual  practice,  the  model  to  be  used  will  need  to  be  based  on 
the  specific  fault  tolerant  characteristics  of  the  design  being 
analyzed.  Models  that  incorporate  system  fault  behavior,  such  as 
shown  in  Figure  2-5,  do  not  specifically  include  SW  as  a  part  of 
the  model.  However,  the  system  control  processing,  a  software 
based  functionality,  determines  the  model  structure  to  account 
for  system  behavior  under  fault  conditions. 

The  reliability  estimates  which  result  from  the  use  of  system 
reliability  models  that  account  for  fault  detection,  isolation, 
and  recovery  are  less  optimistic  than  estimates  from  reliability 
models  based  only  on  the  quantity  of  hardware  supplied  and 
required.  The  reliability  of  the  system  structure  being  modeled 
is  usually  very  sensitive  to  the  total  fault  coverage  provided  by 
the  system  design.  System  designs  that  feature  well -designed 
fault  detection  and  isolation  coupled  with  rapid  and  effective 
recovery  of  system  services  avoid  most  sudden  losses  of  system 
services  due  to  undetected  latent  failures  in  backup  equipment  or 
due  to  the  inability  of  backup  equipment  to  successfully  restore 
system  services  when  failures  to  the  primary  equipment  occurs. 
Similarly,  models  of  HW/SW  systems  that  include  SW  as  well  as  the 
fault  tolerance  characteristics  of  the  system  design  are 
sensitive  to  the  overall  effectiveness  of  the  fault  detection, 
isolation,  and  recovery  provided  by  the  hardware  and  software 
designs. 


2 . l . 2 . 2 . 2 . 2  HW/SW  Systems 

Inclusion  of  software  into  hardware  reliability  redundancy  models 
further  increases  the  complexity  of  the  models.  As  in  the 
hardware-only  reliability  models,  accurate  modeling  of  system 
behavior  requires  that  fault  coverage  (Cd,  Ci,  Cr)  be  included 
into  the  model.  Similarly,  software  fault  coverage  and  the 
impact  of  long  persistence  faults  must  be  included  in  the  system 
models  where  appropriate.  This  results  in  each  model  of 
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redundant  HW/SW  elements  being  uniquely  tailored  to  the  design 
being  analyzed. 

The  examples  of  specific  models  given  below  are  presented  to 
allow  a  skilled  analyst  to  determine  the  system  attributes  that 
need  to  be  considered  in  deriving  a  reliability  model  for  a  set 
of  redundant  elements.  There  is  no  attempt  to  fully  develop  all 
of  the  possible  modeling  situations  that  may  apply  to  a  HW/SW 
system.  Most  of  the  reliability  models  that  will  be  needed  to 
evaluate  the  reliability  of  HW/SW  systems  will  be  specific  to  the 
exact  hardware  and  software  design  being  evaluated.  Only  a 
minimal  level  of  generalization  is  possible. 

2. 1.2. 2. 2.1  Cold_Standbv  Systems 

Redundant  hardware/software  systems  that  use  cold  standby 
techniques  to  provide  fault  tolerance  can  be  modeled  without 
undue  difficulty  as  long  as  automatic  switchover  and  startup 
schemes  are  not  used  in  the  design.  In  general ,  only  the 
hardware  and  software  failure  rates  for  the  HW/SW  elements  need 
to  be  considered  in  developing  the  reliability  model.  For 
designs  that  use  manual  restoration  of  system  services  through 
the  activation  of  an  unpowered  backup  unit,  an  adaptation  of  the 
reliability  model  shown  in  Figure  2-6  can  be  used  to  estimate  the 
reliability  of  the  redundant  structure.  The  models  shown  in  the 
figure  are  based  on  the  earlier  work  of  James  et  al.  (1982) .  As 
shown  in  the  figure,  structure  state  transitions  are  caused  by 
either  hardware  or  software  failures.  Hardware  failures  cause  a 
transition  to  a  state  with  one  less  hardware  element  and 
commencement  of  repair  actions  on  the  failed  element  if  repair  is 
allowed.  The  reliability  model  of  Figure  2-6  does  not  allow 
latent  failures  in  the  backup  element  to  be  modeled.  The  model 
assumes  that  failuras  of  unpowered  elements  are  impossible. 
Similarly,  problems  in  recovering  system  services  are  not  modeled 
since  the  recovery  of  system  services  must  be  directly  managed  by 
the  system  operator.  Software  failures  result  in  system  recovery 
using  the  same  processing  hardware  and  a  restart  of  the  failed 
software.  Both  repairable  and  non-repairable  systems  are  allowed 
to  have  software  restarted  to  enable  recovery  from  software 
failures.  Inclusion  of  the  transition  path  allowing  recovery 
from  software  failures  is  optional  for  non-repairable  systems. 
The  existence  or  lack  of  this  transition  path  will  depend  on  how 
the  equipment  is  operationally  employed. 
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(A)  Repairable  system 


^HW 


RHW 


(B)  Non-Repalrable  system 

Figure  2-6.  HW/SW  Reliability  Model:  M  Required  of  N  Supplied  Identical  Elements,  Cold  Standby 


2. 1.2. 2. 2. 2. 2  Hot  ..Standby.  Systems 

Reliability  modeling  of  hot  standby  HW/SW  systems  requires 
consideration  of  hardware  failure  rates,  software  failure  rates, 
hardware  fault  detection,  isolation,  and  recovery  coverage, 
software  failure  detection,  isolation,  and  recovery  coverage, 
hardware  repair  rates  and  software  restart/recovery  rates.  The 
effect  of  long  persistence  software  failures  on  the  reliability 
achieved  by  hot  standby  redundant  structures  is  included  in  the 
software  fault  coverage  estimates  for  recovery  coverage  (Cr)  . 
Depending  on  the  system  design  being  modeled,  all  or  most  of 
these  parameters  will  be  used  to  help  identify  states  and/or 
transition  rates  between  structure  states.  The  exact  state 
diagrams  that  result  from  an  FMEA  of  the  HW/SW  system  will  depend 
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on  the  design  being  evaluated.  An  example  of  a  reliability  model 
for  a  very  simple  system  structure  is  discussed  below. 

Figure  7.-1  presents  a  simplified  state  diagram  for  a  HW/SW 
structure  with  one  of  two  identical  elements  required.  The  model 
shown  is  for  a  hot  standby  system  with  automatic  switchover. 
This  type  of  structure  is  very  common  in  air  traffic  control, 
command  and  control,  and  air  defense  systems  where  a  high 
probability  of  continuous  system  services  is  required.  In 
modeling  this  structure,  five  parameters  of  interest  are 
recognized.  The  model  states  depend  on  primary  HW  platform  state 
(operational  or  failed) ,  primary  SW  state  (operational  or 
failed) ,  backup  hardware  platform  state,  backup  software  state, 
and  recovery  status.  Recovery  status  is  defined  to  have  two 
states,  successful  or  failed.  A  successful  recovery  indicates 
that  the  structure  has  successfully  transitioned  from  primary 
equipment  to  the  backup  equipment  after  failure  of  either  the 
primary  hardware  or  software.  Alternatively,  successful  recovery 
can  indicate  that  a  failure  in  a  backup  equipment  was 
successfully  detected,  allowing  repair  of  the  backup  equipment  to 
commence.  A  failed  recovery  indicates  that  either  recovery  from 
primary  to  backup  equipment  has  failed  or  that  a  failure  has 
occurred  in  the  backup  equipment  which  has  not  been  detected. 

Since  there  are  five  parameters  of  interest,  each  of  which  has 
two  possible  values,  a  total  of  32  possible  states  would  be 
expected.  However,  soma  of  the  32  possible  states  cannot  exist 
in  practice.  Also,  some  of  the  states  that  can  exist  are 
functional  duplicates  that  can  be  merged.  For  example,  a  state 
with  a  hardware  failure  in  the  primary  equipment  and  operational 
software  in  the  primary  equipment  can  by  shown  to  be  one  of  the 
32  possible  states.  However,  the  state  is  impossible  because 
software  cannot  be  operational  on  a  failed  hardware  platform. 
The  two  states  that  can  exist  for  (1)  a  failed  backup  equipment 
with  successful  recovery  and  (2)  a  failed  primary  equipment  with 
successful  recovery  can  be  shown  to  be  functionally  equivalent 
since  successful  recovery  implies  that  whichever  hardware  remains 
operational  has  been  assigned  to  primary  processing  as  a  part  of 
the  recovery  process. 


23 


1901M 


11 

*  \\ 

1 4_ll 

i 

SSi?- 


Jim  ji  j 

1  m  8  1  8  I  !  I  8  !  i  8  I  1  II 

M  id  M  id 


8 ijs  8 8? 8  !  »J  8  8  8  1 

&  N  &  1  W  a. 


*f  <?  fi?  ff 

sS  s.  s  a 


&  ^  fir  ^  c  ^  s*  5*  w  ?  ^  ^  ?  cr  ^  ^  ^ 

^  £  is£d8£fi£s!lc£jii2Li!££s£i£5s£5££i£c£c£s£$£8.£ 


©o 


a  « 

g  s| 

5  g| 

gl 

f 

is 


•  i  i  ..  .. 

O  U.  I  w  u. 


*»  h  <  a  6  o 


Figure  2-7.  Simplified  Slate  Diagram  for  1:2,  Hot  Standby,  HW/SW  System 


For  the  nodal  of  Figure  2-7,  a  total  of  ten  states  result,  with 
the  following  definitions: 


State  0:  Success  State  -  Fully  Operational  State 

State  1:  Success  State  -  Backup  has  a  detected  SW  failure  which 

is  being  recovered  fron. 

State  2:  Success  State  -  System  is  operational  with  a  latent  SW 

failure  in  the  backup  element. 

State  3:  Success  state  -  System  is  operational  with  a  detected 

hardware  failure  in  the  backup  element. 

State  4:  Success  State  -  System  operational  with  a  latent  HW 

failure  in  the  backup  element. 

State  5:  Failed  State  -  Primary  SW  has  failed,  recovery  to  the 

backup  HW  and  SW  has  not  been  success¬ 
ful.  System  operations  intervention 
will  be  required  to  restore  system 
operation  on  either  HW  platform. 

State  6:  Failed  State  -  Primary  HW  has  failed.  The  recovery 

process  has  failed.  Either  incorrect 
detection,  isolation,  or  incomplete 
recovery  has  occurred.  Manual  inter¬ 
vention  by  the  system  operator  will  be 
required  to  restore  system  services  on 
the  backup  equipment. 

State  7:  Failed  state  -  Software  failures  have  occurred  on  both 

primary  and  backup  system  elements. 

State  8:  Failed  State  -  The  primary  HW  and  backup  SW  have 

failed.  Both  elements  are  down, 
recovery  is  not  possible  without  manual 
intervention  by  the  system  operator 
and/or  maintenance  personnel. 

State  9:  Failed  State  -  Both  hardware  elements  have  failed. 
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As  shown  in  the  figure,  transitions  between  states  occur  due  to 
either  failures  in  the  hardware  or  software  or  due  to  the  status 
of  tha  recovery  process.  Using  state  diagrams  that  model  the 
impact  of  hardware,  software,  and  fault  coverage  for  both 
hardware  and  software  failures  results  in  more  accurate 
approximations  of  the  potential  reliability  of  redundant  systems. 
Also,  accurate  models  that  reflect  the  system  design  decisions 
which  have  been  made  provide  a  basis  for  evaluating  the 
reliability  demands  of  candidate  architectural  approaches  early 
in  the  design  process  [Goddard  (1989)]. 


2.1.3  System  .Model  Evaluation 

State  diagrams  that  have  been  developed  to  model  the  reliability 
of  combined  HW/SW  structures  tend  to  be  relatively  complex. 
Manual  solutions  are  possible,  but  not  considered  practical. 
Numerous  programs  are  currently  available  in  the  commercial 
marketplace  that  can  aid  in  the  solution  of  closed  form  semi- 
Markov  state  diagrams.  Additionally,  government  procurement 
offices  and  their  contractors  have  access  to  both  simulation  and 
analytic  tools  which  allow  solutions  to  these  reliability  models 
through  appropriate  Rome  Laboratory  and  NASA  offices. 
Acquisition  of  one  or  more  of  these  tools  should  be  considered 
before  attempting  to  accurately  model  HW/SW  systems. 


2.1.4  Eatlmfltlqn.  ot  Model . Parameters 

Reliability  models  of  HW/SW  systems  are  based  on  hardware  failure 
rates,  hardware  repair  rates,  software  failure  rates,  software 
recovery  times,  and  the  probability  of  fault  detection, 
isolation,  and  recovery  for  both  hardware  and  software  (hardware 
and  software  coverage) .  Values  for  each  of  the  model  parameters 
will  need  to  be  derived  as  a  part  of  the  reliability  estimating 
process. 

2 . 1 . 4 . 1  Hardware  Failure  Rate  Estimation 

Hardware  failure  rates  for  use  in  combined  HW/SW  models  should  be 
obtained  from  the  same  sources  as  those  traditionally  used  for 
hardware  only  reliability  models.  In  service,  field,  reliability 
records  are  the  best  estimators  of  expected  hardware  failure 
rates.  When  field  reliability  records  are  not  available, 
reliability  test  results  are  the  next  best  estimator  of  expected 
hardware  reliability  performance.  When  neither  field  nor  test 
reliability  records  are  available,  MIL-HDBK-217  is  the  preferred 
reference  for  obtaining  component  level  failure  rates  that  can  be 
used  to  predict  hardware  reliability  performance.  When  MIL-HDBK- 
217  predictions  cannot  be  applied  due  to  lack  of  equipment 
definition,  the  reliability  performance  of  the  previous 
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generation  of  similar  equipment  cav»  be  used  as  an  estimate  of  the 
lover  bound  for  the  expected  reliability  of  current  generation 
equipment. 

2. 1.4. 2  Hardware  Repair  Rate  Estimation 

Hardware  repair  rates  for  use  in  combined  HW/SW  models  should  be 
obtained  from  the  same  sources  as  those  traditionally  used  for 
hardware  only  reliability  models.  In  order  of  preference,  field 
repair  time  data,  maintainability  demonstration  test  data,  MIL- 
HDBK-472  predictions,  and  prior  generation  of  equipment 
maintainability  performance  should  be  used  to  estimate  the 
equipment  repair  time  required. 

2. 1.4. 3  Hardware,  and  Software  Coverage. Estimation 

Fault  coverage  estimates  for  use  in  combined  HW/SW  models  can  be 
obtained  from  a  combination  of  FMEA  to  assess  hardware  built-in 
test  (BIT)  effectiveness,  and  fault  tree  analysis  (FTA)  to  assess 
the  result  of  software  failure.  Assessing  the  likelihood  for  a 
failure  to  be  detected  and  to  leave  the  system  in  a  state  where 
fault  isolation  and  recovery  are  possible  will  require  an  in 
depth  understanding  of  at  least  the  part  of  the  software  design 
dedicated  to  system  control  and  to  the  handling  of  faults.  The 
determination  of  software  fault  coverage  and  software  BIT 
effectiveness  is  an  area  where  further  research  will  be  required 
to  detail  adequate  and  accurate  methods  of  estimation. 

2. 1.4. 4  so ftwttr.fi,  £flsa,vflgy  lima 

Software  recovery  time  estimates  for  use  in  combined  HW/SW  models 
can  be  derived  using  the  methods  of  system  performance  analysis. 
Accurate  estimation  of  these  times  will  depend  on  the  hardware 
available  for  processing,  the  loading  on  the  system,  and  the 
impact  of  the  failure  on  system  resources  (eg.  files) .  The 
results  of  FMEAs  and  FTAs  performed  to  support  estimation  of 
hardware  and  software  coverage  should  provide  an  estimation  of 
the  amount  of  system  resource  damage  that  may  be  attributable  to 
a  given  software  failure.  Estimation  of  software  recovery  time 
during  early  design  phases,  before  test  data  is  available,  will 
need  to  be  based  on  performance  analysis  results.  Further 
research  to  develop  methodologies  for  estimating  system 
performance  metrics  under  various  failure  and  recovery  conditions 
is  needed. 

2. 1.4. 5  Software  Failure  Rates 

Software  failure  rates  to  be  used  in  modeling  combined  hardware 
and  software  systems,  although  constant,  differ  from  hardware 
failure  rates  in  their  dependency  on  operational  profile  and 
workload.  For  purposes  of  illustration,  a  computer,  containing 
one  or  more  individual  processors,  which  is  part  of  a  system,  is 
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used  along  with  Its  software  to  provide  one  or  more  system 
functions.  The  rate  at  which  these  assigned  system  functions  are 
unavailable  is  dependent  on  the  failure  rate  of  the  hardware  and 
the  failure  rate  of  the  software.  Calculating  failure  rates  for 
hardware  elements  was  discussed  in  2. 1.4.2  above.  The  software 
failure  rate  can  be  decomposed  into  three  parts:  the  operating 
system  or  executive  failure  rate,  the  failure  rate  associated 
with  any  re-used  software,  and  the  failure  rate  for  any  newly 
developed  software.  These  software  failure  rates,  once  converted 
to  a  common  time  frame  of  reference  are  additive.  A  reliability 
block  diagram  for  a  hypothetical  processor  is  shown  in  Figure  2- 
8,  below. 
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Figure  2-8.  Reliability  Block  Diagram  for  a  Processor 


Failure  rates  for  operating  systems  or  executives  can  usually  be 
obtained  from  the  supplier  of  the  operating  system  or  executive. 
Failure  rates  obtained  from  the  operating  system  supplier  are 
usually  quoted  in  the  number  of  outages  caused  over  some  period 
of  time  (e.g.,  a  year).  Failure  rates  for  operating  systems  are 
generally  quoted  with  respect  to  system  operating  time  because 
the  operating  system  is  active  at  all  times  when  the  computer  is 
powered  and  ready  for  processing.  The  reliability  analyst  will 
need  to  convert  the  failure  rate  given  to  failures  per  hour  for 
compatibility  with  hardware  failure  rates.  Operating  system 
failure  rates  can  be  substantial  and  should  not  be  ignored. 
Operating  systems  for  mainframe  computers  can  be  several  million 
lines  of  code  in  size  and  are  often  very  complex  and  difficult  to 
completely  debug.  Smaller  computers,  including  single  board 
processor  applications,  sometimes  use  real-time  executives  which 
can  still  contribute  substantially  to  the  overall  software 
failure  rate.  Failure  rates  for  re-used  code  can  be  obtained 
from  applications  where  the  code  was  previously  used.  These 
failure  rates  should  generally  be  much  lower  than  the  failure 
rates  for  newly  developed  code.  The  availability  of  this  data 
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depends  on  the  completeness  of  organizational  record  keeping  \nd 
the  amount  of  code  modification  that  has  been  necessary  to  allow 
the  code  re-use.  If  the  failure  rate  for  re-used  code  is 
available  in  terms  compatible  with  conversion  to  failures  per 
system  operating  hour,  the  failure  rate  can  generally  be  used 
directly  in  the  reliability  modeling.  If  the  failure  rate  for 
re-used  code  is  known  in  failures  per  CPU  operating  period 
(second,  minute,  hour,  etc),  the  failure  rate  will  have  to  be 
converted  to  failures  per  system  operating  hour  using  the  methods 
discussed  in  paragraph  2. 1.4. 5. 3  below. 

Estimates  of  the  failure  rate  for  newly  developed  software  are 
obtained  using  the  research  results  discussed  in  section  4  of 
this  report.  The  failure  rate  estimates  produced  by  these 
methods  is  provided  in  failures  per  CPU  operating  second  for  each 
software  element  being  developed.  These  failure  rates  must  then 
be  combined  as  discussed  in  paragraph  2. 1.4. 5. 4  to  account  for 
the  specific  software  topology  and  timing.  Additionally,  the 
resultant  software  failure  rate  must  be  converted  to  a  system 
operating  hour  form  as  discussed  below  in  paragraph  2. 1.4. 5. 3. 

For  the  purposes  of  developing  software  failure  rates,  a  software 
system  can  be  viewed  as  a  hierarchy.  The  hierarchy  consists  of 
modules,  computer  software  components  (CSCs) ,  computer  software 
configuration  items,  and  so  on,  as  described  in  DOD-STD-2167A. 
Because  of  the  different  names  for  items  at  different  levels,  two 
generic  names  will  be  used  tc  represent  software  items  at 
adjacent  levels  in  the  discussion  that  follows.  The  first 
generic  name  is  " component. "  A  component  is  an  item  at  any  level 
in  the  system  hierarchy.  The  second  name  is  "aggregate."  An 
aggregate  is  an  item  composed  of  an  interrelated  set  of 
components.  An  aggregate  lies  at  the  next  level  up  in  the  system 
hierarchy  from  its  components.  For  example,  an  aggregate  of 
CSCIs  can  be  a  software  subsystem.  The  software  components  that 
comprise  an  aggregate  will  be  related  to  one  another  in  two  ways: 
a  particular  timing  configuration  and  a  particular  reliability 
topology. 

Timing  configuration  describes  the  time  intervals  during  which 
the  various  components  are  active  and  inactive  during  a  period  of 
interest.  Reliability  topology  is  concerned  with  the  number  of 
components  in  the  aggregate  that  can  fail  before  the  aggregate 
fails. 


2 . 1 . 4 . 5 . 1  Timing  Configurations 

Several  different  timing  configurations  are  possible.  The  major 
timing  relationships  between  software  components  are  concurrent 
and  sequential.  Components  will  be  termed  concurrent  if  they  are 
active  simultaneously.  The  components  are  sequential  if  they  are 
active  one  after  the  other.  A  single  component  alternated 
between  active  and  inactive  periods  will  be  termed  intermittent, 


29 


and  several  components  alternating  their  activity  will  be  termed 
interleaved.  Hybrid  timing  configurations,  for  instance  concur¬ 
rent/sequential,  may  also  occur. 

Sc.ne  common  examples  of  timing  configurations:  In  most 
electronic  equipment,  all  hardware  components  are  energized  and 
operating  all  the  time.  The  components  are  therefore  in  a 
concurrent  timing  configuration.  Software  components  (in 
computer  science  called  "processes”  or  "tasks")  execute  in  an 
interleaved  manner  on  a  single  central  processing  unit  (CPU) . 
The  software  and  the  computer  that  hosts  it  are  active 
concurrently.  In  a  multiprocessing  system  with  three  processors, 
the  three  programs  running  on  those  processors  are  active 
concurrently.  In  a  distributed  system,  the  local  and  remote 
hardware  and  the  local  and  remote  software  are  active 
concurrently.  In  a  batch-oriented  computer  system,  the  programs 
execute  sequentially. 

Software  reliability  topology  can  be  expressed  in  the  form  "k- 
out-of-n,"  where  n  is  the  number  of  components  and  k  is  the 
number  that  must  succeed'.  There  are  two  distinguished  cases:  An 
n-out-of-n  configuration  is  the  most  common  and  is  called  a 
series  configuration;  a  l-out-of-n  configuration  is  called  a 
parallel  configuration. 

2. 1.4. 5. 2  Notation 

Capital  letters  will  be  used  to  refer  to  the  aggregate  item  and 
lowercase  letters  to  refer  to  the  component  item.  The  aggregate 
reliability  function  will  be  denoted  R(t)  and  the  aggregate 
failure  rate  will  be  denoted  A(t)  or  A.  The  corresponding 
component  reliability  figures  of  merit  are:  the  component 
reliability  function,  denoted  r(t) ■exp[-At] ,  :md  the  component 
failure  rate,  denoted  1. 

One  reason  that  aggregate  failure  rate  A(t)  can  be  a  function  of 
time  is  that  an  aggregate  of  redundant  constant-failure-rate 
components  will  have  a  time-dependent  failure  rate;  as  the 
redundant  components  fail,  the  aggregate  "ages."  Another  reason 
is  that  a  component  might  not  be  active  during  the  entire  period 
of  interest.  When  the  component  changes  state  from  active  to 
inactive  or  vice  versa,  the  failure  rate  of  the  aggregate  will 
change . 


2 . 1 . 4 . 5 . 3  Failure  Rate  Adjustment 

A  computer  program's  failure  rate  can  be  expressed  with  respect 
to  three  different  time  frames  of  reference:  execution  (CPU) 
time,  system  operating  time,  and  calendar  time. 

A  program  can  only  fail  when  it  is  running.  The  failures  uncover 
faults,  and  the  removal  of  the  faults  results  in  reliability 
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growth.  Thus,  software  reliability  growth  curves  are  based  on 
cumulative  execution  time  and  express  a  single  program's  failure 
rate  in  terms  of  execution  time.  Such  a  failure  rate  is 
sensitive  to  processor  speed:  The  execution-time  failure  rate  is 
always  linearly  proportional  to  the  processor  speed.  If  the 
failure  rate  X  for  a  software  component  was  measured  on  or 
predicted  for  a  processor  with  average  instruction  rate  g,  (e.g., 
instructions  per  second),  and  the  target  machine's  processor  has 
average  instruction  rate  g2,  then  the  failure  rate  should  be 
adjusted  [Musa  et  al.  (1987) j  to 

-  &X  (l) 
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During  the  operation  of  a  system,  programs  may  not  operate 
continuously.  For  example,  some  of  the  programs  might  time-share 
a  single  CPU.  Also,  multiple  CPUs  may  be  present,  allowing 
program  executions  to  overlap.  In  order  to  combine  the  failure 
rates  of  the  various  programs  with  one  another  to  arrive  at  an 
overall  software  failure  rate,  it  is  first  necessary  to  translate 
all  the  program  failure  rates  into  a  common  time  frame  of 
reference.  This  frame  of  reference  is  system  operating  time. 

To  convert  an  execution-time  failure  rate  to  a  system-operating¬ 
time  failure  rate,  the  program's  utilization  utt  needs  to  be 
determined.  This  is  the  ratio  of  execution  time  to  system 
operating  time.  The  utilization  can  exceed  100%  if  the  same 
program  executes  on  more  than  one  CPU  (each  CPU  reading  a 
different  input  stream) . 

The  formula  for  converting  a  program's  execution-time  failure 
rate  At  to  the  system-operating-time  failure  rate  Xt  is 

X,  M  X.-  u„  (2) 

Conversely,  the  execution-time  failure  rate  can  be  recovered  as 

*•  ■  -k 


If  the  programs  are  in  a  series  configuration,  then  the  (average) 
failure  rate  is  simply  the  sum  of  the  system-operating-time 
failure  rates  of  the  individual  programs.  This  result  can  be 
derived  as  follows.  Suppose  there  are  N  software  components  that 
run  during  the  time  period  T.  Let  Ai  be  the  execution-time 
failure  rate  for  the  i-th  software  component.  Let  m(T)  be  the 
expected  number  of  failures  during  that  period.  The  expected 
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number  of  failures  contributed  by  the  i-th  software  component  is 
(u,T) .  Thus 

\i(T)  •  (4) 

The  overall  failure  rate  is 

<*> 

The  sum  EA,u,  is  seen  to  be  the  sum  of  the  programs'  system- 
operating-time  failure  rates. 


2. 1.4. 5. 4  Reliability  Combination  Models 

This  section  is  organized  by  type  of  aggregate.  The  type  of 
aggregate  is  a  conjunction  of  timing  configuration  and  reliabil¬ 
ity  topology.  Table  2-1  shows  the  various  combinations  and 
designates  each  model  by  a  letter.  The  material  presented 
discusses  both  the  software  reliability  calculation  and 
cumulative  failure  rate  model  for  the  timing  configuration  and 
reliability  topology  being  discussed.  The  cumulative  failure  rate 
discussion  is  presented  for  use  in  calculation  of  software 
failure  rates  for  use  in  much  of  the  system  modeling  previously 
discussed. 


Model  Des¬ 
ignation^ 
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F 
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Reliability 


Series 


Series 


Series 


Series 


Series 


Parallel 


k-out-of-n 


Parallel 


Parallel 


Timing 


General 


Indeterminate 


Sequential 


Random 


Concurrent 


Concurrent 


Concurrent 


Concurrent 


Sequential 


Special  Name 


Mission  Model 


Snmi-Markov 


N-Modular  Redundancy 
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The  failure  rate  and  reliability  function  of  a  software  aggregate 
can  in  many  cases  be  expressed  as  a  closed-form  function  of  the 
component  failure  rates  and/or  reliability  functions.  The 
following  discussion  focuses  on  those  cases. 


The  most  co^unon  reliability  topology  for  a  software  aggregate  is 
the  series  configuration.  In  this  configuration,  the  aggregate 
fails  if  and  when  any  one  of  its  components  fails.  The  failure 
rate  of  the  aggregate  is  the  sum  of  the  failure  rates  of  the 
active  components.  Suppose  the  components  are  numbered  1,  2, 

. ..,  k.  Let  the  function  a,(t)  evaluate  to  unity  if  component  i 
is  active  at  time  t,  and  let  it  evaluate  to  zero  otherwise.  Then 
the  aggregate  failure  rate  A(t)  is 

k 

A(t)  =  VA^tt) 
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where  is  the  failure  rate  of  the  i-th  component. 

The  failure  rate  is  useful  when  aggregating  to  higher  levels  and 
is  a  reliability  figure  of  merit  in  its  ran  right.  If  the  result 
is  a  constant  failure  rate,  or  can  be  approximated  by  a  constant 
failure  rate,  then  the  rules  given  here  can  be  applied  to  the 
aggregate  failure  rates  to  find  the  failure  rates  and  reliability 
of  even  higher-level  aggregates. 

2. 1.4. 5. 4. 2  Model  (B)  Mission  Reliability  Model 

A  mission-oriented  system  is  described  by  means  of  a  mission 
profile  and  consists  of  N  consecutive  time  periods,  called 
phases.  During  each  phase  the  mission  has  to  accomplish  a 
specified  task.  An  example  of  a  space  vehicle's  mission  phases 
is  ground  operation,  launch,  and  orbit.  Furthermore,  at  any 
point  in  time  the  system  is  in  one  of  M  possible  operational 
modes.  The  effective  operating  time  Xj  for  the  j-th  operational 
mode  is  given  by 


s 

X-i  *  ^  ]  ^i^ij  •  j -1 , 2 , ... ,  M 


(7) 


where  ts  is  the  duration  of  the  i-th  mission  phase  and  is  the 
fraction  of  time  the  j-th  mode  is  utilized  during  that  phase. 

Suppose  there  are  S  components  in  the  software  aggregate.  Let 
ujk  oe  the  utilization  ("duty  cycle")  of  component  k  during 
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operating  mode  j .  Then  the  amount  of  time  component  k  is  active 
during  the  mission  is 

T,j'm£xJUJJc,  *=1,2,...,W  (8) 


In  matrix  notation ,  the  foregoing  equations  are 


X  «  TZ;  T'  -  XU 


(8) 


(Note  that  T'  is  merely  the  uppercase  of  active  time  r';  the 
prime  does  not  mean  matrix  transpose.)  Let  Xk  be  the  failure 
rate  of  the  k-th  component.  Assuming  that  all  components  form  a 
series  configuration,  the  expected  number  of  failures  from  the  k- 
th  component  is 

\kk  •  ,  ic-1,2,... ,5  (10) 


and  the  expected  number  of  failures  for  a  given  mission  is 


Me.)  -|>*  Ul) 


Note  that  in  practical  situations  these  n  values  will  all  be 
fractional . 

Because  it  is  unknown  which  components  are  active  at  any  moment, 
it  is  not  possible  to  determine  the  instantaneous  failure  rate 
A(t) .  However,  the  average  failure  rate  can  be  computed  as 


A 


•  V#  " 


M(ta) 


(12) 


In  situations  where  the  activation/inactivation  times  are  known, 
it  will  be  possible  to  determine  A(t)  for  all  t.  Paragraphs 
2. 1.4. 5. 4. 3  through  2. 1.4. 5. 4. 5  discuss  some  common  timing 
situations  in  series  configurations. 
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vW*L:l 


penes  aeouentiai 


In  this  situation,  components  1  through  k  are  active  one  after 
the  other.  For  software,  this  arrangement  is  commonly  seen  in 
batch-oriented  systems.  The  time  t(  is  the  point  at  which 
component  i  finishes  and  component  (i+1)  is  activated. 

For  te[tt,t|4l]  the  failure  rate  of  the  aggregate  is 

A (t)  -  Xi+l  (13) 


Sometimes  the  components  are  not  active  consecutively;  a  time 
period  during  which  no  component  is  active  can  be  represented  by 
a  pseudocomponent  whose  failure  rate  is  zero.  If  a  component  is 
active  intermittently,  that  is,  for  several  piecewise  continuous 
periods,  then  a  pseudocomponent  (with  the  same  failure  rate)  can 
be  created  for  each  such  period. 

2. 1.4. 5. 4. 4  Model  (D)  Semi-Markov 

If  the  software  system  consists  of  k  components  that  follow  one 
another  sequentially  according  to  known  probabilities,  then  a 
semi-Markov  technique  [Littlewood  (1979b),  Cheung  (1980), 
Siegrist  (1988)]  can  be  employed  to  obtain  an  overall  failure 
rate  for  the  aggregate. 

A  Markov  process  is  a  stochastic  process  in  which  the  future 
development  is  completely  determined  by  the  present  state.  It 
does  not  matter  at  all  the  way  in  which  the  current  state  arose. 
A  Markov  chain  is  a  Markov  process  that  has  a  discrete  state 
space.  A  semi-Markov  process  [Ldvy  (1954),  Smith  (1958)]  is  a 
Markov  chain  in  which  the  amount  of  time  spent  in  each  state 
before  a  transition  occurs  (the  "sojourn  time")  is  random. 

Let  pf,  be  the  probability  of  the  operation  of  component  j 
following  the  operation  of  component  i.  For  software,  this 
transition  consists  of  software  component  i  passing  control  to 
software  component  j.  The  equilibrium  probabilities  (also  called 
"limiting"  or  "steady-state"  probabilities)  are  the  probabilities 
of  being  in  each  of  the  states  when  the  aggregate  is  operated  for 
a  long  time.  The  equilibrium  probabilities  are  found  by 
solving  the  system  of  equations 


Jc 

rtj  •  .  i*l,2,...,k  (14) 


together  with  Em,=l. 
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Let  mn  be  the  mean  time  that  component  i  is  active  before  it 
finishes  and  is  followed  by  component  j  (the  "sojourn  time") . 
Then  the  failure  rate  of  the  aggregate  is  approximated  by 


A 


k  k 


1  >1 


k  k 


£  ^  niPiJmU 


(15) 


Failures  can  also  occur  during  the  transition  from  one  component 
to  another.  If  r,,,  the  reliability  of  the  transition  from 
component  i  to  component  j  is  considered,  then  the  m,,!,  in  the 
numerator  can  be  replaced  by  m^Aj+fl-r^)  to  take  transition 
failure  into  account. 

2. 1.4. 5. 4. 5  Modal  (E)  Series__CQncugrent 

If  throughout  a  time  interval  [0,t#],  components  1,  ...,  k  are 

active,  then  the  failure  rate  at  time  te[0,t#]  is 

A(t)-f;X1  (1«) 


If  all  k  components  have  the  same  failure  rate  1,  the  aggregate 
failure  rate  will  be 


A 


■  kX 


(17) 


2 . 1 . 4 . 5 . 4 . 6  Redundant  Software  Configurations 

Redundancy  is  the  provision  of  additional  software  components 
beyond  the  minimum  needed  to  perform  the  functions  of  the 
software  aggregate.  The  purpose  of  this  redundancy  is  to 
increase  the  reliability  of  the  aggregate.  Unlike  hardware,  all 
copies  of  a  computer  program  are  identical  and  will  all  fail 
under  the  same  circumstances,  so  simple  duplication  of  the 
software  is  of  limited  benefit  due  to  the  potential  for  long 
persistence  software  failures  which  were  discussed  previously. 
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Software  redundancy  is  accomplished  by  providing  multiple 
distinct  versions  of  the  software  developed  to  the  same  require¬ 
ments  specification.  The  goal  is  for  it  to  be  very  unlikely  that 
multiple  versions  will  fail  on  the  same  inputs.  By  having  each 
version  independently  developed  by  a  different  programming  team, 
the  hope  is  that  the  versions  will  turn  out  differently  enough 
that  they  will  not  often  fail  on  the  same  inputs. 

Multiversion  programming  is  a  controversial  topic.  Knight  and 
Leveson  (1986)  showed  through  a  large-scale  experiment  that 
independently  developed  versions  do  not  necessarily  fail 
completely  independently  of  one  another.  In  fact,  the  number  of 
coincident  failures  was  "surprisingly  high."  Eckhardt  and  Lee 
(1985)  developed  a  detailed  statistical  model  of  multiversion 
programming.  "Independently  developed  programs"  are  modeled  as 
programs  randomly  selected  from  the  universe  of  possible  program 
versions  that  purport  to  solve  the  problem  at  hand.  The  indepen¬ 
dence  criterion  is  enforced  by  requiring  the  joint  probability  of 
selecting  the  programs  to  equal  the  product  of  the  marginal 
probabilities  of  selecting  each  one. 

Eckhardt  and  Lee,  who  concentrated  on  modeling  N-version  program¬ 
ming,  express  the  reliability  of  the  N-version  aggregate  as 

r  - 1  -  f  ue> 


where  m«(N+l)/2,  a  majority  of1  the  N  versions.  The  aggregate 
fails  whenever  at  least  m  versions  fail.  6(x)  is  the  proportion 
of  versions  failing  when  executing  on  input  state  x.  Q (A)  is  the 
usage  distribution,  the  probability  that  the  subset  of  input 
states  A  is  selected. 

One  of  Eckhardt  and  Lee's  major  results  was  that  independently 
developed  versions  do  not  necessarily  fail  independently.  Up 
until  the  Eckhardt  and  Lee  model  came  out,  practitioners  of 
multiversion  programming  had  set  independent  development  as  their 
goal.  Eckhardt  and  Lee  show  that  the  departure  from  independent 
behavior  is  governed  by  Var{9),  where  6-0 (X),  where  X  is  a 
randomly  chosen  input.  If  this  probability  0  is  identical  for 
all  the  programs,  then  Var(0)-O  and  the  independently  developed 
programs  fail  independently;  otherwise  the  reliability  will  be 
overstated  by  Var{0).  The  best  candidates  for  multiversion 
programming  are  those  problems  for  which  Var{6)  is  low. 
Intuitively,  Var{9)  is  the  variation  in  difficulty  in  processing 
different  inputs.  Different  independently  developed  versions 
will  tend  to  fail  on  the  same  "hard"  input  cases,  even  though  the 
different  versions  use  different  solutions.  The  independently 
developed  versions  do  not  necessarily  fail  via  similar  faults  or 
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output,  but  they  will  nevertheless  tend  to  fail  on  the  same 
inputs . 


Littlewood  and  Miller  (1989)  built  on  the  Eckhardt  and  Lee  model 
to  show  that  forced  diversity  can  improve  reliability  to  a  point 
where  it  is  superior  to  independent  failure  behavior.  Diversity 
is  modeled  by  design  decisions  that  induce  a  partition  on  the 
universe  of  possible  programs.  Much  more  research  needs  to  be 
done  in  how  to  practically  inject  diversity  and  how  to  quantify 
the  reliability  benefits  of  such  diversity.  Since  chance  alone 
does  not  guarantee  the  independence  of  the  multiple  versions, 
diversity  must  somehow  be  deliberately  built  into  the  software; 
this  is  also  a  topic  for  research. 


In  an  active  parallel  aggregate,  all  components  operate  simulta¬ 
neously  and  all  must  fail  for  the  system  to  fail.  This 
reliability-related  use  of  the  word  "parallel"  should  not  be 
confused  with  its  use  in  the  computer  science  term  "parallel 
processing. " 

The  parallel  configuration  is  useful  when  the  components 
continuously  perform  some  "chore."  It  is  not  useful  in 
situations  where  the  components  produce  an  "answer."  Software 
and  some  types  of  hardware  produce  information.  If  the 
components  differ  in  the  answer  produced,  there  is  no  provision 
in  a  parallel  system  for  reconciling  the  different  answers. 
Other  techniques  such  as  recovery  blocks  and  N-modular 
redundancy,  do  provide  a  means  of  deciding  which,  if  any,  answer 
to  consider  correct. 


If  k  components  are  active  simultaneously  and  all  components  must 
fail  for  the  system  to  fail,  then  the  aggregate  reliability  is 

*  k 

R(t)  ■  1-J[J  (l-exp [-A.J  t] )  -  l-JJ  [1-^(0  ]  (19) 


The  failure  rate  [Nieuwhof  (1975) ]  is 


(20) 
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where 


a ,  « 


*  TTJt) 


,  i»l,  2 , Jc 


(21) 


In  the  case  of  the  same  failure  rate  A  for  each  component,  then 


R(t)  -  l  -  (i-exp[-X  t]  )*  ■  1  -  [r  (t)  ]  * 


(22) 


and  [Grosh  (1982)] 


A(t)  ■  kkBAt) 


(23) 


where 


b  / >.\  ,  exp  [-A.  t]  ( 1  -exp  [ -k  fc] )  * 
*  1- (1-exp  [-X  fc]* 


(24) 


Note  that  the  failure  rate  of  the  aggregate  Is  not  constant  but 
Increases  monotonically  through  time  and  converges  to  k. 
Intuitively,  the  reason  is  that,  as  time  goes  on,  the  redundant 
components  will  eventually  succumb  one-by-one  to  random  failure, 
causing  the  level  of  redundancy  to  degrade.  However,  this  assumes 
that  individual  software  components  are  not  restarted  upon 
failure 

The  effective  aggregate  failure  rate  (RADC  (19e8) ]  is  a  const  ut 
approximation  based  on  time  to  first  failure: 


A  ■ 


(25) 


2. 1.4. 5. 4. 6. 2  Model  k-out-of-n  Configuration 

A  k-out-of-n  aggregate  consists  of  n  components  such  that  the 
aggregate  succeeds  if  and  only  if  k  of  its  components  succeed. 
When  the  components  have  differing  failure  rates,  computation  of 
the  reliability  function  is  more  an  algorithm  than  a  formula. 
The  best  way  to  calculate  the  reliability  function  of  such  a 
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system  is  through  a  computer  program  such  as  described  and  listed 
in  Sarje  and  Prasad  (1989).  The  most  common  situation,  where  the 
components  have  equal  failure  rates,  will  be  considered  here. 

If  k  out  of  n  components  with  the  same  failure  rate  must  succeed 
for  the  aggregate  to  succeed,  the  reliability  is 


R(t)  (a«) 


After  R(t)  is  computed,  the  failure  rate  can  be  obtained  as  [Rau 
(1970)] 


A(t) 


R(t) 


(27) 


The  effective  aggregate  failure  rate  [RADC  (1988)]  is  a  constant 
approximation  based  on  time  to  first  failure: 

A  -  — — 

A  1  (28) 


2. 1.4. s. 4. 6. 3  Modal ..(H)  M-Mpduiftg  Rtdundancy 

In  an  aggregate  with  N-modular  redundancy,  N-2n+l  redundant 
components  are  active  concurrently.  A  voter  compares  the  results 
from  the  N  components  and  chooses  the  results  of  the  aggregate  as 
the  output  of  a  majority  of  the  components.  If  there  is  no 
majority,  the  aggregate  fails.  Furthermore,  for  the  aggregate  to 
succeed,  the  majority  must  have  the  correct  output.  The 
aggregate  reliability  is  the  same  as  that  of  an  (n+1)  -out-of-N 
aggregate : 


R(t) 


£  (  [r(  t)  ] i  [1-r  ( t)  ]  ^ 

J-n* l  \  J  / 


(29) 


The  effective  failure  rate  [RADC  (1980)]  is  a  constant  approxima 
tion  based  on  time  to  first  failure: 
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N  i 

E  7 

i-n*  1 


(30) 


An  important  caveat  is  that  N-modular  redundancy  can  result  in 
worse  reliability  than  the  component  alone  [Dhillon  and  Singh 
(1981)].  For  example,  for  N«3— triple  modular  redundancy  (TMR) — 
the  reliability  of  the  aggregate  is 

R(t)  -  [r(t)]a[3-2r(t)]  (31) 


Solving  R(t)-r(t)  yields  a  crossover  point  of  At»0.693.  Before 
the  crossover  point,  TMR  will  have  superior  reliability;  after 
that  point  the  simplex  component  will  be  better.  In  a  mission- 
oriented  system  the  mission  time  should  be  such  that  X<0.693. 

The  use  of  N-modular  redundancy  for  software  fault  tolerance  is 
called  N-version  programming  [Avizienis  (1980)].  The  reader  is 
referred  to  the  Eckhardt  and  Lea  (1985)  modal  for  insights  into 
the  limitations  of  multiversion  programming.  The  formulas  given 
in  this  section  apply  when  it  can  be  justified  that  the  multiple 
versions  fail  independently. 

2. 1.4. 5. 4. s. 4  Maflu  (i)  standby  Redundancy 

Software  standby  redundancy  is  a  scheme  in  which  k  redundant 
components  are  tried  in  sequence  until  one  succeeds  or  all 
components  have  been  exhausted.  The  aggregate  only  fails  if  and 
when  all  coxnponents  have  failed. 

The  reliability  [Grosh  (1989)]  of  the  aggregate  is 


The  effective  failure  rate  is  a  constant  approximation  based  on 
time  to  first  failure: 
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In  the  usual  case  of  all  components  having  ths  a am*  reliability, 


Hit) 


exp  t-1 t] 


JJ-1 


£ 


Ut) i 
it 
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The  formula,  based  on  the  Erlang  distribution,  assumes  perfect 
sensing  and  switching,  and  no  idle  failure  rate.  If  the 

reliability  of  the  switcher  is  R  ,  then  all  terms  in  the 
summation  except  the  first  must  be  weighted  by  Rau: 


R(t)  ■  exp  [-It] 


r„  _  w  *.w(Xfc)a 
1  +  RtwX  t+— - 


Rw  (U)* 


k! 


<3ii) 


The  failure  rate  of  the  aggregate  [Becker  and  Jansen  (1977)]  is 


A(t) 


_ X _ 

<*-l)  Ig  (X  t)  i 


(30 


The  failure  rate  increases  over  time  and  slowly  approaches  the 
constant  failure  rate  X.  The  effective  failure  rate  [RADC 
(1988)]  is  a  constant  approximation  based  on  time  to  first 
failure: 


A  - 


X 

k 


(37) 


Software,  standby  redundancy  is  implemented  through  the  technique 
of  recovery  blocks  [Randall  (1975)].  The  results  of  each 
component  version  are  checked  through  an  acceptance  test.  The 
acceptance  test  may  itself  be  susceptible  to  failure.  The 

aggregate  now  can  fail  in  three  circumstances  [Scott  at  al. 

(1987)]:  (E,)  An  incorrect  result  is  accepted;  (E2)  A  correct 

result  is  not  accepted;  and  (E3)  An  incorrect  result  is  not 
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accepted.  suppose  that  the  acceptance  test  has  reliability  rT 
and  therefore  failure  probability  qT-l-rT.  Each  component  has 
failure  probability  q»l-r. 

The  formulas  given  here  apply  when  it  can  be  justified  that  the 
multiple  versions  fail  independently.  To  simplify  the  equations, 
let  X-q.-2qTq4-q.  If  the  reliability  of  the  acceptance  test  is 
the  probability  of  the  first  type  of  aggregate  failure  occurring 

is 


Pr  {Elt  n } 


qT-q'(l-Xn) 

l-X 


(38) 


The  probability  of  the  second  type  of  aggregate  failure  occurring 
is 

PrfE^n}  -  (qT-qTv)  ‘  XB'1  (39) 

and  for  the  third  type  of  aggregate  failure  it  is 

?i[e3,  n }  -  (q-q^q)  *  x^1  (40) 


The  reliability  of  the  aggregate  is  then  calculated  as 

3 

Rn  -  1-g  Pr  {Eit  13}  (41) 


If  the  recovery  software  is  imperfect  and  exhibits  reliability 
rR,  an  error  type  E4  of  unsuccessful  recovery  can  occur.  For  the 
case  of  n-1  independent  versions  of  the  software,  Pr{E4,l}«0;  for 
n-2  the  probability  is 


PrU"4,2}  »  r‘(l-rT)  Tjj  +  (1-r)  *rT\r*  (42) 


For  n^3,  the  recurrence  relationship  is 

pr  {E4,n}  ■  Pr  {Ei,n-D  +Pr  {£4,  n-2)i>  [r*  (l-rr)  +  (l-r)  rT]  (43) 
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The  reliability  of  the  aggregate  is  then  obtained  as 


R 


n 


1 


Pr  {Eit  n} 


(44) 


2. 1.4. 5. 4. 7  Combination  Concurrent-Sequential  Configuration 

To  handle  an  aggregate  that  contains  both  concurrent  and 
sequential  components,  divide  the  time-line  into  time  segments 
such  that  one  or  more  components  are  concurrently  active 
throughout  each  such  time  segment.  For  each  time  segment, 
replace  the  components  with  a  pseudocomponent.  Let  the  failure 
rate  of  the  pseudocomponent  be  the  sum  of  the  failure  rates  of 
the  concurrently  active  components  in  that  time  segment.  Once 
this  "collapsing"  of  time  segments  is  completed  for  all  time 
segments,  the  aggregate  can  be  treated  as  purely  sequential. 

2. 1.4. 5. 4. 8  combination  Series-Parallel  Configuration 

If  a  system  consists  of  series-connected  parallel  configurations, 
first  reduce  each  parallel  configuration  to  a  single 
pseudocomponent,  and  handle  as  a  series  system.  If  a  system 
consists  of  parallel -connected  series  configurations,  first 
reduce  each  series  configuration  to  one  pseudocomponent,  and 
handle  as  a  parallel  system.  These  strategies  can  be  applied 
recursively  to  lower  levels,  until  the  component  level  is 
reached. 
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3.  RELIABILITY  DECOMPOSITION  AND  ALLOCATION 


Reliability  allocation  is  a  planning  technique  for  guiding  design 
and  implementation  toward  meeting  specified  reliability 
requirements.  It  is  especially  useful  when  different  teams  or 
subcontractors  work  on  different  parts  of  a  system.  In  system 
functional  decomposition,  the  functional  requirements  of  the 
system  are  allocated  to  subsystems  and  lower-level  items.  In 
reliability  allocation,  the  overall  system  reliability 
requirements  are  apportioned  to  those  same  items. 

In  fulfillment  of  SOW  Task  2,  techniques  were  developed  for 
decomposing  hardware/software  systems  and  allocating  quantitative 
reliability  requirements  to  the  hardware  and  software  subsystems 
and  to  lower  indenture  levels. 

3.1.  Appro ash 

Reliability  allocation,  in  its  simplest  form,  consists  of  finding 
an  achievable  combination  of  failure  rates  and  repair  rates  that 
supports  achievement  of  a  system's  or  subsystem's  specified 
reliability  requirements.  One  needs  to  start  with  a  model,  such 
as  obtained  from  the  modeling  techniques  of  Section  2,  and 
"solve"  for  the  unknown  failure  rates  and  repair  rates.  The 
failure  rates  and  repair  rates  cannot  have  just  any  values;  the 
values  are  constrained  by  the  range  of  values  that  are 
realistically  achievable.  The  reliability  engineer  bagins  the 
allocation  process  by  determining  what  the  achievable  range  is 
for  each  rata.  An  acceptable  allocation  is  one  in  which  all 
rates  lie  within  their  achievable  ranges  and  the  overall 
subsystem  or  system  reliability  meets  or  exceeds  the  specified 
reliability  requirement.  The  reliability  modal  is  evaluated  to 
assess  whether  the  allocated  values  meet  or  exceed  the  specified 
requirements.  The  process  of  finding  the  failure  rates  and 
repair  rates  involves  a  systematic  trial-and-error  exploration  of 
alternatives.  The  reliability  engineer  makes  an  initial  cut  by 
choosing  rates  from  the  achievable  ranges.  The  values  can  be 
chosen  arbitrarily  but  ideally  would  be  chosen  based  on  intuition 
and  on  experience  with  similar  and  previous-generation  items. 
Next,  the  reliability  is  evaluated  through  the  reliability  model. 
If  the  allocated  values  (MTBF,  MTTR,  etc)  ensure  that  the  system 
requirements  are  met,  the  allocation  is  complete  and  any  "excess" 
can  be  either  allocated  to  another  part  of  the  system,  or 
reserved  as  a  means  of  mitigating  risk. 

For  software  components,  timing  relationships,  relative  execution 
frequency,  complexity,  and  criticality  are  important 
considerations.  Several  techniques  were  developed  for 
apportioning  reliability  requirements  and  goals  through  the 
levels  of  a  software  hierarchy.  One  of  these,  "Allocation  Based 
On  Achievable  Failure  Rata"  is  a  bottom-up  methodology.  The  other 
allocation  methods  presented  are  top-down  methodologies. 
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3.2.  Results 


The  trade-off  between  implementing  functionality  in  hardware  or 
software  often  depends  on  cost  considerations:  Software 
development  costs  can  be  amortized  over  each  unit  produced,  but 
hardware  has  a  per-unit  manufacturing  cost.  Once  the  system 
functionality  is  partitioned  into  hardware  and  software 
subsystems,  the  levels  of  software  hierarchy  are  established 
through  DOD-STD-2167A  and  a  software  design  methodology  (such  as 
data  structure-oriented  design,  data  flow-oriented  design,  or 
object-oriented  design) .  The  levels  of  software  decomposition, 
such  as  computer  software  components  (CSCs)  and  computer  software 
configuration  items  (CSCIs) ,  refer  to  parts  of  the  static  program 
as  it  is  viewed  for  the  purposes  of  configuration  management. 
When  executing  the  software  subsystem  will  exhibit  a  dynamic 
structure — the  timing  relationships  and  reliability  topologies  of 
different  threads  of  execution.  Reliability  allocation  should 
take  place  at  the  level  at  which  individual  threads  of  execution 
("processes"  or  "tasks")  exist.  Generally,  this  level 
corresponds  to  the  CSCI  level.  This  level  is  also  appropriate 
for  allocation  because  the  interfaces  among  modules  are  included. 


3.2.1.  saXtwacft-  RtliafalllAy.„AUQfiat.lflD 

If  the  utilization  factors  for  each  CSCI  being  allocated  to  are 
known  or  can  be  estimated,  the  bottom-up  method  "Allocation  Based 
On  Achievable  Failure  Rate"  can  be  used.  When  utilization  factors 
for  the  CSCIs  cannot  be  estimated,  one  of  the  top-down 
allocations  methods  should  be  used.  When  using  these  top-down 
allocation  methods,  several  iterations  may  be  required  to  ensure 
that  the  allocations  to  the  individual  components  are  achievable 
and  fully  support  the  system  requirements. 

3.2.2  Notation  for  Software  Reliability  Allocation  Models 

The  term  "aggregate"  refers  to  an  interrelated  collection  of 
software  "components."  In  systems,  the  aggregate  is  a  software 
subsystem  as  a  whole,  and  a  component  is  a  CSCI  within  that 
subsystem.  In  a  distributed  system,  the  aggregates  of  one 
computer  system  can  serve  as  the  components  of  the  overall 
distributed  system,  and  so  the  allocation  would  be  performed 
twice,  once  to  allocate  from  the  distributed  system  as  a  whole  to 
the  individual  computer  systems,  and  once  to  allocate  from  the 
individual  computer  systems  to  their  respective  processes 
(generally,  CSCIs). 

Aq:  Failure  rate  goal  for  a  software  aggregate. 

R„(t) :  Reliability  goal  for  a  software  aggregate.  Note  that 
AQ»(-ln  R„(t)]/t  and  R0(t)-exp[-A0t] . 
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Ac:  Failure  rate  objective  for  a  software  component. 

rc(t) :  Reliability  objective  for  a  software  component. 

Let  Rg  be  *_he  reliability  goal  for  a  mission.  The  purpose  of 
reliability  allocation  is  to  set  reliability  objectives  (rs)v 
(r0)2/  . ..,  (rG)n  for  components  1,  2,  . ..,  n  so  that 

£  i  (fg)  2  >  •  •  •  •  ( jj)  i  Rg  ( 45  J 


where  f  is  defined  by  the  rules  of  reliability  combination 
covered  previously.  An  infinite  number  of  different  allocations 
can  satisfy  the  above  inequality.  A  good  allocation  not  only 
meets  or  exceeds  the  overall  system  reliability  requirements  but 
optimizes  or  "balances"  the  allocation  in  some  way. 

3.2.3  Allocation  Techniques 

The  allocation  techniques  that  are  presented  here  are  Allocation 
Based  on  Achievable  Failure  Rates,  Equal  Apportionment, 
Proportional  Allocation,  Weighted  Allocation,  Constrained 
Allocation,  and  Re-allocation.  Appendix  C  covers  the  "Dynamic 
Allocation”  class  of  re-allocation  techniques. 

The  bottom-up  allocation  method,  "Allocation  Based  on  Achievable 
Failure  Rates"  requires  the  ability  to  estimate  CSCI  utilization 
rates  and  the  test  time  available  for  each  CStl.  This  method 
provides  a  set  of  failure  rate  allocations  to  each  CSCI  which  are 
achievable  within  the  planned  program  test  schedule  and  which 
accurately  reflect  the  planned  usage  and  the  execution  time 
available  for  achieving  reliability  growth.  If  failure  rate 
allocations  provided  by  this  method  do  not  support  achievement  of 
the  system  or  subsystem  specified  reliability  figures  of  merit, 
it  is  an  indication  that  there  may  be  a  problem  with  the 
specification  requirements  or  one  or  more  higher  level 
allocations  with  respect  to  the  program  schedule. 

In  "Equal  Apportionment,"  the  components  of  an  aggregate  are 
allocated  equal  failure  rates  or  equal  failure  probabilities  in 
such  a  way  that  the  aggregate  meets  its  reliability  goal. 

"Proportional  Allocation"  trkes  into  account  the  length  of  time 
each  component  is  active.  The  longer  a  component  is  active,  the 
more  exposure  it  has  to  the  possibility  of  failure.  The  premise 
of  proportional  allocation  is  that  higher  reliability  should  be 
demanded  of  components  that  are  active  for  a  greater  share  of  the 
time  relative  to  the  other  components. 


In  "Weighted  Allocation,"  the  failure  rate  allocated  to  a 
component  is  based  on  the  criticality  of  the  component  and/or 
feasibility  of  its  meeting  a  reliability  objective.  The 


criticality  cf  the  component  includes  the  consequence.;  of  the 
failure  to  mission  success  and  safety.  Tne  feasibility  [Fuqua 
(1986)]  of  ;.i  reliability  objective  is  dependent  on  the  complexity 
(number  of  parts  for  hardware,  program  size  [or  other  measure] 
for  software) ,  the  state-of-the-art,  and  the  skills  and  tools 
available  for  the  development  process. 

In  "Constrained  Allocation,"  the  allocation  is  optimized  with 
respect  to  additional  considerations  such  as  cost. 

In  "Re-allocation,"  a  previous  allocation  is  revised  because  one 
or  more  components  could  not  meet  their  reliability  objectives. 
Dynamic  allocation  (Appendix  C)  is  a  technique  for  re-allocation 
with  minimized  effort.  Table  3-1  summarizes  the  allocation 
models. 

Table  3-1.  Allocation  Models 


Class 

Type  of  Allocation  !  Topology 

Timing 

Bottom  Up 

A 

Achievable  Failure  Rate 

Ail 

All 

Top  Down 

i 

mm 

Eqvial  Apportionment 

Series 

Top  Down 

mm 

Equal  Apportionment 

Series 

Concurrent 

Top  Down 

D 

Equal  Apportionment 

Parallel 

Concurrent 

Top  Down 

E 

Equal  Apportionment 

Parallel 

Sequential 

F 

Equal  Apportionment 

NMR 

Concurrent 

Top  Down 

G 

Proportional  Allocation 

Series 

tBSBSBM 

Top  Down 

H 

Proportional  Allocation 

Parallel 

Concurrent 

Top  Down 

I 

'  nortance  Factors 

Series 

Concurrent 

Top  DCitn 

J 

Weighted  Allocation 

Series 

Concurrent 

Top  Down 

K 

Constrained  Allocation 

Series 

Concurrent 

Top  Down 

_ 

Re-Allocation 

Series 

Concurrent 

Dynamic  allocation  models  appear  in  Appendix  C. 


3.2.3. l  Model  (A)  Allocation  Based  On  Achievable  Failure  Rate 

Allocation  Based  on  Achievable  Failure  Rates  uses  each  CSCT's 
utilization  (see  paragraph  2. 1.4. 5. 3).  The  utilization  governs 
the  growth  rate  a  CSCI  will  experience  during  system  test.  All 
things  being  equal,  the  greater  a  CSCI's  utilization,  the  faster 
its  reliability  will  grow  during  system  test. 


A  forecast  is  made  of  each  CSCI's  initial  failure  rate  by  using 
predicted  size  and  processor  speed  and  assuming  industry  average 
figures  for  fault  density  and  other  prediction  model  quantities 
(see  Chapter  4)  .  Then  the  software  reliability  growth  model 
parameters  are  predicted  to  determine  the  growth  curve  formula. 
An  achievable,  failure  rate  figure  is  obtained  from  the  growth 
curve.  The  relative  distribution  of  achievable  failure  rates  is 
used  to  apportion  the  aggregate's  failure  rate  goal  Ae  to  the 
CSCIs. 

Note  that  changing  to  a  faster  processor  does  not  change  the 
system-operating-time  failure  rate;  the  reduction  in  u,  is  offset 
by  a  proportionate  increase  in  the  execution-time  failure  rate. 
Therefore,  a  CSCI's  allocated  system-operating-time  failure  rate 
does  not  need  to  be  changed  if  the  hardware  platform  changes  to  a 
faster  processor. 

The  software  aggregate's  failure  rate  goal  is  assumed  to  be 
expressed  with  respect  to  system  operating  time. 


The  allocation  scheme  is  based  on  first  determining  an  achievable 
distribution  of  failure  rates  among  the  CSCIs,  considering  their 
predicted  initial  failure  rates  and  how  much  reliability  growth 
they  can  each  expect  to  experience  during  system  test.  The 
distribution  of  failure  rates  is  then  used  to  allocate  the 
software  aggregate's  failure  rate  goal  AQ  to  the  constituent 
CSCIs. 

A  CSCI's  achievable  failure  at  release  is  determined  by 
predicting  an  initial  failure  rate  10  and  applying  a  software 
reliability  growth  curve  to  forecast  CSCI's  failure  rate  at 
release. 

The  i-th  CSCI's  initial  failure  rate  is  the  failure  rate  at  the 
start  of  system  test  (t,»0)  .  Using  the  Proposal/Pre-Contractual 
stage  prediction  model  (see  section  4.),  a  CSCI's  initial  failure 
rate  with  respect  to  execution  time  is 

X0  *  r  •  K  '  «0/J  (46) 

where  r  is  the  processor  speed  (average  number  of  instructions 
executed  per  unit  of  time) ,  K  is  the  fault  exposure  ratio 
(4.20xl0'r  industry  average),  uQ  is  the  CSCI's  fault  content 
t developed  XLOC  x  6  industry  average),  and  I  is  the  CSCI's  number 
of  object  instructions).  The  number  of  object  instructions  is 
found  by  multiply'  g  the  number  of  predicted  source  instructions 
bj  the  programming  language's  code  expansion  ratio  (see  paragraph 
4.2.2  and  Table  4-2).  For  example,  the  code  expansion  ratio  for 
Ada  is  4.5.  Note  that  the  initial  failure  rate  is  primarily 
determined  by  fault  density  and  processor  speed,  as  opposed  to 
size. 
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It  is  thus  assumed  that  each  CSCI  is  developed  by  a  mature, 
reproducible  software  development  process  that  produces  code,  with 
the  industry-average  of  6  faults  per  developed  KLOC  (developed 
excludes  reused  code) . 

The  growth  curve  relates  failure  rate  decline  to  cumulative  wall- 
clock  time  expended  in  system  test: 

“  *0  *  expE-p^uJ  •  U*  (47) 

where  0  is  the  decrement  in  failure  rate  per  failure  experienced: 

P  -  5  .  it  (48) 


An  industry-average  value  of  0.955  is  recommended  for  B.  B  is 
the  fault  reduction  factor,  the  average  net  number  of  faults 
removed  per  failure.  The  cumulative  wall-clock  time  expended  in 
system  test  t^  may  differ  from  CSCI  to  CSCI  if  the  software  is 
incremented  in  a  series  of  successive  "builds."  If  the  software 
is  incremented  all  at  once,  then  the  t^'s  will  be  equal. 


It  is  assumed  that  the  CSCIs  are  in  series,  so  the  CSCI  failure 
rates  with  respect  to  system  operating  time  must  add  up  to  the 
overall  software  failure  rate  Once  the  achievable  failure 
rates  ^(t),  ^(t),  ...,  ^(t)  are  determined,  each  CSCI  is 
assignees  a  relative  weight 


"i 


( fci) 


(4S) 


These  relative  weights  add  up  to  one.  The  failure  rate  iUj,  with 
respect  to  system  operating  time,  to  be  allocated  to  the  i-th 
CSCI  is 

^Ai  "  Ao  '  W J  (SO) 


The  allocation  is  acceptable  since  the  CSCI  failure  rates  add  up 
to  the  goal: 


53  "  Ao53  wi  "  Aa  *  1  u  a0 


(51) 


To  change  a  CSCI’s  allocated  system-operating-time  failure  rate 
to  execution  time,  the  allocated  failure  rate  is  divided  by  the 
CSCI ' s  utilization  u^. 
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(52) 


Let  the  failure  rate  goal  of  the  aggregate  be 

A  ,  -ln*e<t> 


The  strategy  of  equal  apportionment  ia  for  the  system  never  to  be 
exposed  to  an  instantaneous  failure  rate  exceeding  Aq.  This 
results  in  an  overall  reliability  of  at  least  expC-Agt]-!^. 

The  formulas  for  equal  apportionment  are  obtained  by  solving  the 
corresponding  reliability  combination  formulas  for  component 
failure  rate  or  reliability. 


3. 2. 3. 2.1  Motel  (B)  . affluent iol  .Series 

Each  sequentially  active  component  is  allowed  a  failure  rate  of 
Aq-Aq,  so  that  a  constant,  satisfactory  failure  rate  is 
maintained  at  all  times  regardless  of  which  component  is  active. 


3 . 2 . n . 2 . 2  Model  (cn  Concurrent Series 


Each  of  k  concurrently  active  components  is  allocated  a  failure 
rate  objective  of  A.^JL/k.  This  results  in  an  overall 
reliability  of  at  least 


exp 


RaU) 


(53) 


The  reliability  of  each  component  is 

r  ( t)  « 


(54) 


Alternatively,  consider  the  failure  probability  (unreliability) 

Qa(t)  -  1  -  <ss> 


Each  component  can  be  allocated  an  equal  portion  of  the  failure 
probability  for  a  reliability  objective  of 


ra(t)  -  1 


wVt) 

k 


(56) 
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If  k  components  are  connected  in  parallel,  the  failure  rate  1G  to 
be  allocated  to  each  component  is 


K 


(57) 


A  similar  problem  is  to  determine  how  many  components  with 
reliability  r  are  needed  to  meet  an  aggregate  reliability  goal  of 
Rq.  The  solution  is 


_  ln(l-J*a) 
*  ln(l-r> 


(58) 


3. 2. 3. 2. 4  Model  JE)  Standby  Redundancy 

For  recovery  blocks  (under  the  assumption  that  the  multiple 
versions  fail  independently)  with  k  components,  the  reliability 
allocation  is 


ra(t)  -  l  - 


(59) 


3. 2. 3. 2. 5  Modal  (F)  N-Modular  Redundancy 


For  N-version  programming  (under  the  assumption  that  the  multiple 
versions  fail  independently) ,  the  failure  rate  allocation  is 


A.0 


(50) 


3. 2. 3. 3  Proportional  Allocation 


Let  the  expected-number-of-f allures  objective  be 

Mg  -  -In  [*„(«:)]  (61) 


The  strategy  of  proportional  allocation  is  for  each  component  of 
the  aggregate  to  contribute  the  same  expected  number  of  failures. 
The  expected  number  of  failures  is  a  function  of  the  failure 
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rates  and  the  tine  exposures  of  the  components  that  comprise  the 
aggregate . 


3 . 2 . 3 . 3 . 1  Model  i'G>  Sequential  Series 


For  each  of  the  k  components,  let  the  expected-number-of-f allures 
allocation  be 


(62) 


and  allow  each  component  i  a  failure  rate  of 

o  m  "•^7  ,  .  .  .  ,k  (63) 

ti 


where  t, '  is  the  amount  of  time  component  i  is  active, 
results  in  an  overall  reliability  of  at  least 

k 

expt-gX^Cj]  -  Ra 


This 

(64) 


The  reliability  objective  for  component  i  is 

r^t)  -  expMX^a  t]  ,  i«l,2,...,fc  (65) 


Thus  the  longer  the  time  exposure  of  the  component  the  lower  its 
failure  rate  must  be.  This  criterion  is  appropriate  for  hardware 
and  for  software  that  is  run  repetitively. 

3. 2. 3. 3. 2  Modal  (H)  Concurrent  Parallel 

For  proportional  allocation  of  k  components  connected  in 
parallel,  the  reliability  objective  to  be  allocated  to  each 
component  is  [Dennis  (1974)] 


r0(t)  -  l  -  k/r-R~0(  O' 


(66) 
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and  the  failure  rate  objective  ie 

,,  s  -In  [l-kji-RgUf] 
aa)i  .  - Y_2 - 

ti 


i- 1,2, .  . . ,  Jc 


(«7> 


3. 2. 3. 4  Moflami  insartansfl  Eagtoci 

Let  k  be  the  number  of  subsystems;  n,,  the  number  of  items  in 
each  subsystem  i;  R,  the  desired  mission  reliability;  and  t, ' , 
the  mission  time  for  subsystem  i.  Let 

k 

w-gn,  («a> 


be  the  total  number  of  components  in  the  system.  Assign  each 
subsystem  i  an  importance  index  w,  that  is  the  probability  that 
the  mission  will  fail  if  the  i-th  subsystem  fails.  The  failure 
rate  objective  to  be  allocated  to  the  i-th  subsystem  [AGREE 
(1957)]  is 


i )  G 


InEl-l/wj  (l-Rgi/y)  ] 
-t  i 


«9) 


Software  fault  tree  analysis  [Leveson  and  Stolzy  (1983)]  is 
useful  for  pinpointing  safety-  and  reliability-critical  sections 
of  computer  programs.  However,  software  fault  tree  analysis 
examines  the  software  at  too  low  a  level  to  provide  quantitative 
criticality  figures  which  can  be  used  in  developing  importance 
indices.  Regular  fault  tree  analysis,  which  stops  at  the 
interfaces  to  the  software  components,  is  capable  of  providing 
quantitative  criticality  figures. 

3. 2. 3. 5  Model  (J)  Weighted  Allocation 

If  the  mission  reliability  goal  is  R.,  then  the  unreliability  (l- 
Rg)  can  be  allocated  proportionally  to  each  phase  in  the  mission 
according  to  the  distinct  conditions  at  each  stage  [Amstadter 
(1971)].  If  the  weighting  of  phase  i  is  with 

^  <*>i  ■  1  (70) 


then  the  reliability  objective  for  that  phase  is 
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(Ra)d  - 


(71) 


and  the  failure  rate  objective  ie 

Ua)  1  -  Aa  •  ^  (72) 

3. 2. 3. 6  Model  (K)  Constrained  Allocation 

A  software  system  consists  of  n  serial  stages.  A  reliability  of 

<7S) 


is  allocated  to  each  of  the  n  stages.  The  minimum  number  of 
redundant  components  at  each  stage  i  [Sandler  (1963)]  is 


‘  In  (l-.Rg)  ‘ 
lnh-x*) 


(74) 


If  c,  is  the  cost  of  components  for  the  i-th  stage,  the  lnast- 
cost  allocation  is  given  (Moecovitz  and  McLean  (1956)  ] 


In  (1-f^1) ' 

in 


(78) 


where 


_ 

In  [l-xjT 


In  ( 


£2 _ ) 

1  ~Tj)  j 


(76) 
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3. 2. 3. 7  Modal  tl*\  Re-AllQCatlflJl 


Suppose  Ag  is  ths  failure  rate  goal  and  At  is i  thi  currant 
reliability  of  the  component.  Than  if  A„  is  less  than  A.,  a 
suitable  reliability  goal  1,  for  each  component  [von  Alven 

(1964)]  is 


A* 

X 


(77) 


It  is  most  effective  to  raise  the  reliability  of  the  least 
reliable  component  in  a  series  configuration,  -  ana  the  most 
reliable  component  in  a  parallel  configuration. 

Appendix  C  develops  and  presents  "dynamic  allocation"  techniques 
for  minimizing  the  effort  required  for  reallocation. 


/ 

/ 

/ 

i 

/ 

/ 

I 
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4.  SOFTWARE  RELIABILITY  PREDICTION 

Reliability  prediction  has  several  important  uses.  Early  in  the 
acquisition  cycle  it  can  be  employed  to  determine  the  feasibility 
of  the  system  reliability  requirements.  It  can  highlight  which 
components  in  a  design  can  be  anticipated  to  contribute  most  to 
the  system  unreliability.  It  snables  scfe-Vile  and  cost  forecasts 
to  he  made.  It  allows  tradeoffs  to  b<>  between  reliability 
and  other  factors  (such  as  maintain -V;  Ucy- .  Often,  absolute 
prediction  values  are  not  as  impu.'i.-^it  as  relative  values, 
especially  when  comparing  alternative  configurations  or  designs. 

In  fulfillment  of  SOW  Task  4,  a  technique  for  modeling  and 
predicting  the  reliability  of  hardware/software  systems  was 
developed.  Emphasis  was  placed  on  reliability  prediction  for  the 
software  part  of  the  system,  because  hardware  reliability 
prediction  techniques  are  already  well  established. 

4.1  Approach 

The  goal  of  the  prediction  research  was  the  development  of  a 
method  for  the  prediction  of  software  reliability  that  could  be 
used  in  conjunction  with  MIL-HDBK-217  and  other  published  failurs 
rats  data  to  obtain  a  reliability  prediction  for  the  overall 
system.  Hardware  reliability  prediction  provides  a  failure  rate 
for  the  inherent  reliability,  which,  in  theory,  is  the  best 
reliability  achievable  as  the  dsvelopmcnt  and  manufacturing 
procssses  are  perfected. 

Software  does  not  have  the  same  kind  of  inherent  reliability 
limitation.  The  reliability  of  software  will  generally  improve 
over  time  as  faults  ars  uncovered  through  testing  and  are  fixed. 
There  are  two  conditions  that  limit  the  attainable  reliability 
for  software:  (1)  There  reaches  a  point  at  which  a  program  that 
has  poor  maintainability  becomes  a  victim  of  entropy— the  code 
has  been  modified  sc  much  that  it  has  become  a  patchwork  quilt 
and  an  attempt  to  fix  one  fault  results  in  the  introduction  of  at 
least  as  many  new  faults.  (2)  The  software  achieves  a  high  level 
of  reliability,  so  that  the  times  bstwson  failures  ars 
excruciatingly  long,  with  the  result  that  further  testing  becomes 
impractical.  As  discussed  in  the  section  of  this  report  on 
Reliability  Growth,  the  simultaneous  execution  of  multiple 
identical  copies  of  the  software  can  be  helpful  in  postponing 
that  point  in  time. 

When  software  reliability  or  combined  hardware/software 
reliability  is  predicted,  that  prediction  must  be  related  to  a 
point  in  time.  The  earliest  point  in  time  for  which  it  makes 
sense  to  predict  reliability  is  when  the  system  is  fully 
integrated  and  is  operating  in  an  environment  that  emulates  field 
use.  This  point  in  time  is  the  beginning  of  "system  test"  and  is 
denoted  by  cumulative  operating  time  t=o.  The  software 
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reliability  prediction  methods  developed  in  this  section  will 
forecast  the  reliability  the  software  can  be  expected  to  exhibit 
at  that  point.  Also  forecast  will  be  the  parameters  of  the  time 
domain  software  reliability  growth  model  described  in  Section  5 
of  this  report,  so  that  software  reliability  can  be  predicted  for 
points  in  time  later  during  system  test.  It  is  assumed  that  when 
the  software  code  has  been  frozen  and  ie  being  subjected  to  a 
stationary  operational  profile  (probability  distribution  of  input 
states — see  Section  5) ,  it  will  exhibit  a  constant  failure  rate 
(see  Appendix  B) . 

The  study  team  applied  multivariate  regression  (SOW  para. 
4. 1.3. 2. 3)  and  related  statistical  analysis  to  the  data  collected 
in  the  reliability  database  (SOW  para.  4. 1.1.1).  The  team 
explored  which  product  (requirements,  documentation,  code)  and 
software  development  process  characteristics  are  significantly 
correlated  with  fault  content  (SOW  para.  4. 1.3. 3,  4. 1.3. 2. 3). 

The  technique  for  software  reliability  prediction  uses  metrics 
derived  from  characteristics  of  the  software  development  process 
and  of  the  products.  Also  providsd  are  techniques  for  taking 
into  account  the  influence  of  two  additional  factors  on  the 
reliability  figures:  the  program  structure  and  the  operational 
profile. 

The  study  team  first  investigated  techniques  for  predicting 
program  size  during  the  early  stages  of  software  development, 
before  the  aode  has  been  written  (SOW  para.  4. 1.3.1).  Program 
size  is  important  for  relating  fault  content  to  fault  density, 
and  is  a  product  metric  in  its  own  right. 

4.1.1  PrrUqtlon  p t  So ftazara  slsa  <aQW  para*..  4*  1.3-til 

Halstead's  (1977)  software  science  was  the  first  technique  the 
study  team  evaluated.  Software  science  is  an  information- 
theoretic  approach  to  properties  of  representations  of 
algorithms.  By  invoking  these  properties,  one  can  obtain 
predictions  of  various  quantities  of  interest  such  as  program 
size,  complexity,  and  fault  density.  Surprisingly  little  data  is 
required  for  the 'Predictions. 

Using  software  science,  the  size  of  a  program  can  be  predicted  if 
the  number  of  distinct  operators  (verbs)  and  the  number  of 
distinct  operands  (nouns)  in  the  program  are  known.  Software 
science  defines  the  length  of  the  program  as 

LENGTH  ■  nilog2r)i  +  T)3log2Tlj  (78) 

where  is  the  total  number  of  distinct  operators  und  r?,  is  the 
total  number  of  distinct  operands.  The  phenomenon  that  the  size 
of  a  program  is  a  function  of  the  size  of  its  lexicon  has 
parallels  in  other  fields.  The  linguist  George  Zipf  (1965) 
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observed  this  phenomenon  in  natural  languages  such  as  English , 
and  B.  Mandelbrot,  the  father  of  fractals,  saw  the  phenomenon  in 
unordered  random  symbols  [Jones  (1986)]. 

The  number  of  executable  source  instructions  is  obtained  as 

j.  -  (79) 


where  Q-7  for  higher-order  languages  (HOLs)  and  Q-4.5  for 
assembly  language.  From  no  additional  data,  the  fault  density 
can  also  be  obtained.  The  software  science  "volume"  metric  is 
defined  as 

VOLUME  ■  LENGTH  x  loga  (i^+tlj)  (80) 

The  volume  is  roughly  defined  as  the  number  of  characters  that  it 
takes  to  encode  the  program.  The  number  of  faults  is  predicted 
by 


,,  „  VOLUME 
0  3000 


(81) 


and  the  fault  density  by 


- 


(82) 


The  number  of  distinct  operators  and  the  number  of  distinct 
operands  are  available  after  coding  but  can  usually  be  estimated 
at  the  end  of  detailed  design.  However,  software  soience 
provides  a  means  of  making  the  same  predictions  based  on  data 
obtainable  even  earlier,  during  preliminary  design. 

The  early  prediction  method  is  based  on  the  number  of  input  and 
output  parameters  and  the  "language  level."  Since  module 
interfacee  are  established  during  preliminary  design,  the  number 
of  input  and  output  parameters  for  each  module  will  be  available. 


Let  nj*  be  the  number  of  input/ output  parameters  of  a  module. 
Let  t  be  the  language  level.  Then  the  total  number  of  distinct 
operators,  r?i  and  the  total  number  of  distinct  operands,  r?2/  can 
be  determined  by  simultaneously  solving 


ti2  ■  C <2  +t)2) /2]  (n1-2)/(2+na)  +n \  (83) 


and 
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t  (2+  T|j)  loga  (2+TiJ)]2  -  ( (T\1loglx\i*(]2log2r\2)  log2  <tia  +r|a)  (84) 

A  drawback  to  this  method  is  that,  while  Halstead  theorized  that 
language  level  is  a  function  of  the  programming  language,  it 
appears  that  it  is  also  a  function  of  the  individual  programmer 
[Pressman  (1982)].  If  historical  data  is  available  on  the 
programmer  who  is  expected  to  implement  the  module,  an  estimate 
of  the  language  level  can  be  found  from  taking  a  sample  of  his 
code  and  computing 

(2ti  'f 

— -  (T)iiogat)1+naiogan2)  tiogjCt^+na)]  <®5> 


where  N2  is  the  total  number  of  operand  appearances  in  the  code. 

The  study  team  was  skeptical  of  software  science  because  of 
research  articles  critical  of  the  theoretical  and  empirical  bases 
of  Halstead's  work  [Coulter  (1983),  Hamer  and  Frewin  (1982, 
1985),  Shen  et  al.  (1983)].  The  size  of  already  written  programs 
was  "predicted"  using  mean  language  level  figures.  The  results 
were  mixed.  Trying  out  the  software  science  relationships  on 
already-written  software,  calculating  language  level 
historically,  to  see  if  those  relationships  hold  is  recommended. 

The  next  program  size  method  the  team  investigated  was  function 
point  analysis  [Albrecht  (1979),  Albrecht  and  Gaffney  (1983)]. 
This  method  allows  program  size  to  be  predicted  after 
requirements  analysis,  which  is  even  earlier  than  predictions  can 
be  made  using  software  science.  A  function  point  is  one  end-user 
function.  The  requirements  statements  are  examined  to  determine 
the  number  of  logical  end-user  functions  required,  namely  inputs, 
outputs,  inquiries,  master  files,  and  interfaces.  Dreger  (1989) 
discusses  the  manner  in  which  these  functions  should  be  counted. 
Tho  counts  are  multiplied  by  the  empirically  obtained  factors 
given  in  Table  4-1. 


Table  4-1.  Function  Point  Factors 
Function  Point  Factor 

Inputs _ 4 _ 

Outputs _ 5 _ 

Inquiries _ 4 _ 

Master  Files _ 10 _ 

Interfaces  7 
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The  products  of  the  counts  and  the  factors  are  totaled  and 
adjusted  ±25%  based  on  the  estimated  processing  complexity  of  the 
program.  The  following  characteristics  are  considered  tc 
increase  processing  complexity:  data  communications,  distributed 
data  or  processing,  performance  objectives,  heavily-used 
configuration,  transaction  rate,  on-line  data  entry,  end  user 
efficiency,  on-line  update,  complex  processing,  reusability, 
conversion  and  installation  ease,  operational  ease,  multiple  site 
use,  and  facilitation  of  change.  Dreger  (l'i'9)  provides 
guidelines  for  quantifying  the  complexity  added  by  each  of  the 
characteristics . 

The  translation  of  the  number  of  function  points  to  the  number  of 
lines  of  code  depends  on  the  programming  language.  For  Ada, 
Jones  (1986)  has  determined  that  there  are  approximately  71 
source  statements  per  function  point.  For  JOVIAL,  the  number  is 
106;  for  basic  assembler,  320;  for  macro  assembler,  213;  for 
FORTRAN  and  COBOL,  105;  and  for  C,  128. 

The  use  of  function  point  analysis  in  software  engineering  is 
expanding,  especially  as  a  measure  of  productivity.  The  drawback 
to  function  point  analysis  is  that  it  is  currently  geared  to 
business  applications  areas.  Methods  applicable  to  a  wider 
variety  of  application  areas  were  sought. 

People  in  the  defense  industry  involved  in  making  ‘zing 
predictions  were  interviewed  and  a  comprehensive  comp  M  otive 
evaluation  of  software  sizing  models  [AFCCJ3  (1987)  ]  was  reviewed. 
The  sizing  method  that  emerged  at  the  top  of  the  AFCCE  assessment 
is  that  implemented  by  an  increasingly  popular  commercial  product 
called  Software  Sizing  Model  (SSM)  [Bozoki  (1987)].  This  method 
was  developed  by  Dr.  George  Bozoki  of  Lockheed.  SSM  has  also 
been  incorporated  into  RCA's  computerized  cost-estimating 
software,  PRICE.  The  method  uses  psychological  scaling  methods 
to  convert  qualitative  sizing  information  (relative  sizes  of 
modules)  to  absolute  sizes. 

The  exact  inner  workings  of  SSM  are  not  published,  but  the  study 
team  has  devised  similar  methods  based  on  the  scholarly 
literature.  An  example  of  a  psychological  scaling  method  is  the 
technique  of  paired  comparisons;  one  or  more  "experts" ,  persons 
familiar  with  the  type  of  application  and  who  have  worked  cn 
similar  projects  within  the  same  organization,  are  presented  with 
random  pairings  of  module  names  and  asked  to  judge,  for  each 
pairing,  which  module  they  would  expect  to  be  bigger.  If  at 
least  two  of  these  modules  are  "reference  modules"  of  known  size, 
then  it  is  possible  to  compute  predictions  of  the  absolute  sizes 
of  every  module.  Appendix  D  discusses  some  psychometric  methods 
that  have  been  adapted  for  the  purpose  of  program  size 
prediction. 
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4.1.2 


(SOW  para.  4. 1.3.2) 


A  software  failure  is  a  discrepancy  between  the  program’s  output 
and  that  dictated  by  requirements  [Musa  et  al.  (1987)].  A 
software  fault  is  a  missing,  extra,  or  defective  statement  or  set 
of  statements  that  is  the  cause  of,  or  can  potentially  cause,  a 
software  failure  [Musa  et  al.  (1987)].  Talcing  a  snapshot  of  the 
program  at  the  start  of  system  test,  the  inherent  fault  density 
of  a  program  is  the  fault  content  (number  of  inherent  faults) 
divided  by  the  number  of  executable  lines  of  code.  The  term 
"inherent  faults"  will  be  used  to  contrast  with  faults  that  are 
inadvertently  introduced  during  repair  activity. 

An  industry  average  figure  for  fault  density  is  6  faults  per 
kiloline  of  executable  source  code  [Musa  et  al.  (1987)].  This 
figure  may  be  useful  in  forming  very  early  predictions  of 
software  reliability,  for  example,  in  a  proposal.  For  the 
regression  analysis,  the  team  sought  methods  for  determining  the 
number  of  faults  in  a  program.  Unfortunately,  the  faults  in  a 
program  are  not.  directly  observable.  Any  faults  that  have  been 
identified  have  presumably  already  been  removed.  What  is 
observable  during  system  test  are  software  failures.  Each 
software  failure,  by  definition,  is  caused  by  a  single  fault. 

The  occurrence  of  a  software  failure  is  conclusive  evidence  that 
a  fault  exists.  But  counting  the  number  or  rate  of  failures  over 
time  does  not  by  itself  say  anything  about  the  number  of  faults 
that  remain.  The  only  way  to  count  all  the  faults  in  the  program 
is  to  keep  testing  until  all  faults  have  been  exposed  and 
removed.  This  feat  would  require  exhaustive  testing  of  alJ 
possible  input  states,  which,  for  most  programs,  would  take  an 
astronomical  amount  of  time  to  complete.  (One  round  of 
exhaustive  testing  would  not  even  be  enough,  because  of  the 
possibility  that  repair  activity  will  introduce  new  faults.) 


One  proposed  method  for  estimating  the  number  of  faults  is  called 
"fault  seeding."  It  is  based  on  the  same  idea  as  "capture- 
recapture"  studies  used  in  estimating  the  size  of  animal 
populations.  Such  a  study  proceeds  as  follows:  First  comes  the 
"capture"  phase.  A  group  of  animals,  say,  zebras,  are  caught, 
counted,  tagged,  and  released  back  into  the  wild.  Next  comes 
the  "recapture"  phase.  After  a  period  of  time  the  researchers  go 
out  and  capture  a  group  of  zebras.  The  numbers  of  tagged  and 
untagged  zebras  in  tnis  new  group  are  noted.  The  proportion  p  of 
tagged  zebras  in  the  group  is  calculated.  The  number  of  untagged 
zebras  in  the  entire  population  is  estimated  by  dividing  p  into 
the  number  of  untagged  zebras  in  the  second  group. 

In  software  fault  seeding,  originally  discussed  by  Mills  (1972), 
a  number  of  artificially  created  faults  are  injected  into  the 
code.  Testing  proceeds  and,  at  some  point,  the  number  of  seeded 
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faults  and  indigenous  faults  found  are  noted.  An  estimate  of  the 
number  of  indigenous  faults  in  the  program  is  obtained  from  the 
assumption  that  the  proportion  of  indigenous  faults  found  out  of 
the  population  of  indigenous  faults  is  the  same  as  the  proportion 
of  seeded  faults  found. 


For  software ,  let  y  be  the  number  of  faults  injected  initially; 
u,  the  number  of  indigenous  faults  found  so  far;  v,  the  number  of 
seeded  faults  found  to  date.  Then  the  maximum  likelihood 
estimate  of  the  number  of  indigenous  faults  x  is 

x  -  •  uj  (86) 


[Feller  (1957)]. 

The  problem  that  fault  seeding  encounters  in  practice  is  that  the 
seeded  faults  are  easier  (usually  much  easier  [Musa  et  al. 
(1987)])  to  find  than  the  indigenous  faults.  To  carry  the  animal 
analogy  further,  suppose  that  the  zebras  that  are  originally 
captured  are  "tagged"  by  having  one  leg  amputated.  Then  the 
three-legged  zebras  are  released  into  the  wild.  Later,  for  the 
recapture  phase,  a  group  of  zebras  is  caught  and  the  number  of 
three-legged  zebras  noted.  Obviously  the  method  is  flawed 
because  the  threo-legged  zebras  are  much  easier  to  catch  than  the 
untagged  (four-legged)  zebras. 

Variations  on  fault  seeding  have  been  proposed.  Myers  (1976) 
suggests  having  two  independent  teams  test  the  software.  The 
faults  that  the  two  teams  find  in  common  are  treated  as  if  they 
had  been  tagged.  The  formula  is  [Shooman  (1983)] 

<*>o  *  (87) 


where  B,  is  the  number  of  faults  that  tester  #1  found,  B2  is  the 
number  of  faults  that  tester  #2  found,  and  bc  is  the  number  of 
faults  found  in  common.  This  variation  has  the  advantage  that  no 
artificial  faults  have  to  be  created.  However,  several 
conditions  need  to  be  met.  First,  the  faults  found  during  the 
teat  period  must  be  representative  of  the  total  population  of 
faults.  Second,  the  independent  testing  and  debugging  must 
result  in  similar  program  versions,  and  third,  the  common  faults 
must  be  representative  of  the  total  fault  population.  (It  could 
be  argued  that  the  faults  found  in  common  will  tend  to  be  the 
easiest  ones  to  find.) 

Ideally,  the  dependent  variable  in  the  multivariate  regression  is 
fault  density.  The  study  team  concluded  that  the  state  of  the  art 
in  fault  seeding  is  not  advanced  enough  to  provide  good  estimates 
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of  fault  content.  Since  neither  software  science  nor  fault 
seeding  can  currently  provide  accurate  fault  content  figures,  the 
study  team  decided  to  choose  an  observable  quantity,  the  Basic 
Execution  Time  Model  initial  failure  rate  parameter,  X0,  as  the 
dependent  variable  for  the  regression  analysis.  Additional 
methods  provide  the  parameters  B  and  K.  Established  parametric 
relationships  (sow  para.  4. 1.3.3)  are  then  employed  to  derive 
fault  density,  fault  content,  and  growth  rate  predictions. 
Because  processor  speed  varied  from  project  to  project,  the 
observed  failure  rates  were  standardized  to  an  instruction 
execution  rate  of  3  million  instructions  per  second  (MIPS)  . 

Automated  support  for  the  multivariate  regression  was  provided  by 
the  SAS  (Statistical  Analysis  System)  package.  Custom  software 
to  pre-  and  postprocess  the  data  and  perform  additional 
calculations  was  also  written. 

In  predicting  fault  density,  the  two  product  characteristics  that 
immediately  came  to  mind  were  size  and  complexity.  The  two  most 
commonly  employed  complexity  mearures  are  McCabe's  (1976) 
"cyclomatic  complexity"  and  Halstead's  (1977)  "E". 

To  calculate  McCabe's  metric  a  program  graph  is  drawn  to  depict 
flow  of  control.  The  metric  V(G)  is  computed  a s  the  number  of 
enclosed  areas  on  the  plane  of  the  graph.  The  metric  can  be 
obtained  after  the  detailed  design  is  complete. 

Halstead  "E",  which  denotes  "effort",  is  given  by 

E  (^iiog2ni+n2-i-og^3)  [ioga(tii+na)  1 

2 1\2  (88) 
h  1^2 


where  N1  is  the  total  number  of  distinct  operators  and  N2  is  the 
total  number  of  distinct  operands.  Because  of  the  presence  of 
the  quantity  N2,  this  metric  can  only  be  obtained  after  coding. 

Halstead  even  provides  the  following  formula  to  obtain  the  number 
of  faults  from  the  effort: 


£2/3 

<  •  a  - 

0  3000 


(89) 


Several  studies  have  shown  that  complexity  correlates  with 
program  size  [Shen  et  al.  (1983),  Sunohara  et  al.  (1981),  Basil! 
and  Hutchens  (1983),  Gremillion  (1984)].  For  this  reason  the 
study  team  decided  not  to  collect  a  separate  complexity  metric. 
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4.1.3 


(SOW  para.  4.1.32) 


As  detailed  in  Appendix  A,  this  study  has  hypothesized  24 
characteristics  that  affect  the  software  failure  rate  as  shown 
below  broken  down  by  software  life  cycle  phase. 

Requirements  Analysis: 

-  x1 :  Errors  in  requirements  specification  (SRS) 

-  x2:  Requirements  statements  in  the  SRS 

-  x3:  Pages  in  the  SRS 

-  x4  i  Man-months  for  requirements  analysis 

-  x5:  Requirements  changes  after  baseline' 

Preliminary  Design: 

-  x6:  Errors  in  preliminary  design  documents 

-  x7:  Number  cf  computer  software  components  (CSCs) 

-  x8:  Units  (or  Ada  packages)  in  design  structure 

-  x,:  Pages  in  design  documents 

-  x10:  Man-months  for  preliminary  design 

Detailed  Design: 

-  x„:  Errors  in  design  documents 

-  x12:  Man-months  for  detail  design 

-  xl3:  Design  flaws  identified  after  baseline 

-  x14:  Design  flaws  identified  after  an  internal  review 

Coding:’ 

-  x1s:  Number  of  executable  lines  of  code  (LOC) 

-  x16:  Faults  found  through  cods  reviews 

-  x17:  Programmer  skill  level  (average  years  of  experience) 

-  x18:  Number  of  units  undergoing  review 

-  x19:  Average  number  of  source  LOC  per  unit 

Unit  Test: 

-  x20:  Average  number  of  branches  in  a  unit 

-  x21:  Percent  of  branches  covered 

-  x22:  Nesting  depth  average 

-  x2J:  Number  of  times  a  unit  is  unit-tested 

-  x24:  Man-months  for  coding  and  unit  test 

x25  “  xu  +  xh  +  xi6  (Defects  identified  from  walkthroughs 
and  reviews) 

Some  of  the  independent  variables  were  normalized  based  on  kilo 
lines  of  executable  code  (KLOC) .  The  normalized  independent 
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variables  are  x.,  . . . .  ,xu,x.,,x24,x25.  The  formula  to  normalize  x, 
where  i  «  {1,2, ... ,14,16,24,25}  is 

x,  -  (x,  /  x15)  •  1000  (80) 

Recall  that  x15  i3  the  number  of  LOC  in  each  observation. 

For  each  pnase  of  the  software  development  process,  a  new 

regression  model  was  formed  which  involves  Y  (software  failure 
rate  as  the  dependent  variable)  and  x,,  i-  l,2,...,k  where  k  <5  n, 
and  contains  more  software  characteristics  as  they  become 

available.  Therefore,  several  regression  models  were  formulated 
for  each  software  development  phase  (such  as  Requirements 

Analysis,  Preliminary  Design,  Detailed  Design,  coding  and  Unit 
Test) . 

Scatter  plots  of  Y  (software  failure  rate)  versus  x,  (software 
characteristic),  i  -  l,2,...,n,  based  on  the  collected  data  from 
Appendix  A,  were  drawn.  These  scatter  diagrams  suggest  that  the 
regression  models  can  be  linear  in  terms  of  all  the  variables 
except  x6,  x11#  x.3,  xu,  x16,  xa.  However,  both  linear  and 

nonlinear  regression  models  involving  software  failure  rate 
(dependent  variables)  and  the  software  characteristics 
(independent  variables)  described  above  were  developed  for  each 
phase  of  software  development. 

The  general  linear  regression  model  is  of  the  form 


Y  -  a0  +  a1x1  +  a2x2  +  . . .  +  a^ 


(81) 


where  k  assumes  different  values  depending  on  the  software 
development,  phase.  Similarly,  the  non-linear  regression  model 
for  each  phase  of  the  software  development  is  a  sum  of  the  linear 
combination  of  items  in  a  subset  of  (x1f . . .  ,x9,x7, . . .  ,x10,x12,x17, 
. . .  ,x„,x24,x2s}  and  the  linear  combination  of  items  in  the  subset 
of  (l/x6,  l/xn,  l/x13,  l/x14,  l/x16,  l/xa);  the  sizes  of  these 

subsets  depend  on  the  phase  of  the  software  development.  Notice 
that  x.j  is  not  used  in  any  of  the  regression  models,  except  to 
normalize  appropriate  independent  variables,  since  the  software 


failure  rate 
software. 


Y  should  not  depend  on  the  size  (x1s)  of  the 


Due  to  the  fact  that  the  number  of  metrics  exceeded  the  number  of 
observations  in  all  software  development  phases,  except  for  the 
Requirements  Analysis  phase,  full  regression  models  cannot  be 
formulated.  Advanced  statistical  techniques  were  applied  to 
reduce  the  number  of  independent  variables  which  can  participate 
in  the  regression  model  after  each  phase  of  software  development. 
The  reduction  was  accomplished  in  two  alternative  ways,  one  way 
was  to  work  with  only  the  subset  of  metrics  that  was  determined 
to  most  influence  the  reliability  of  the  software.  This  was  done 
by  using  the  stepwise  selection  method.  The  other  approach  taken 
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was  to  work  not  with  the  metrics  themselves  but  with  aggregates 
of  the  metrics.  Principal  component  analysis  was  employed  to 
achieve  this  goal. 

The  regression  models  that  were  formulated  after  each  software 
development  phase  were  compared  to  one  another  in  the  Validation 
Task.  The  comparisons  were  done  based  on  the  values  of  their 
statistical  characteristics  (such  as:  R2,  p-values,  95% 
confidence  limits,  etc.),  and  on  how  well  they  can  predict  the 
failure  rates  of  known  software.  A  model  was  chosen  to  be  used 
to  describe  the  relationship  between  software  failure  rate  and 
software  characteristics. 

The  following  is  the  discussion  of  the  "stepwise  selection" 
method  and  principal  components  analysis. 


Since  it  would  have  been  impractical  to  try  all  subsets  of 
independent  variables,  the  stepwise  selection  method  was  used  to 
effectively  eliminate  some  independent  variables  which  have  a 
small  contribution  to  the  regression  model. 

Stepwise  regression  is  a  composite  of  two  different  procedures: 
forward  selection  and  backward  elimination. 

The  forward  selection  technique  begins  with  no  variables  in  the 
model.  Then  it  selects  the  independent  variable  that  is  most 
highly  correlated  with  the  dependent  variable.  For  each  of  the 
independent  variables,  forward  selection  calculates  F-statistics 
that  reflect  the  variable's  contribution  to  the  model  if  it  is 
included.  The  F-statistic  is  the  ratio  of  the  explained  variance 
to  the  unexplained  variance.  The  p-values  for  these  F-statistics 
are  compared  to  a  predetermined  value,  f.  If  no  F-statistic  has 
a  significance  level  smaller  than  f,  forward  selection 
terminates.  Otherwise,  forward  selection  adds  to  the  model  the 
remaining  independent  variable  that  has  the  largest  F-statistic 
(accomplishes  the  greatest  reduction  in  unexplained  variance) . 
Forward  selection  then  calculates  F-statistics  again  for  the 
variables  still  remaining  outside  the  model,  and  the  evaluation 
process  is  repeated.  Thus,  variables  are  added  one  by  one  to  the 
model  until  no  remaining  variable  produces  a  significant  F- 
statistic.  Once  a  variable  is  added  to  the  model,  it  stays 
there . 

The  backward  elimination  technique  begins  by  calculating 
statistics  for  a  model,  including  all  of  the  independent 
variables.  Then  the  variables  are  deleted  from  the  model  one  by 
one  until  all  the  variables  remaining  in  the  model  produce  F- 
statistics  significant  at  a  pre-specified  level,  b.  At  each 
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step,  the  variable  showing  the  smallest  contribution  to  the  model 
is  deleted. 


The  stepwise  method  is  a  cross  between  forward  selection  and 
backward  elimination.  The  variables  already  in  the  stepwise 
model  do  not  necessarily  stay  there;  the  decision  is  reversible. 
As  in  forward  selection,  variables  are  added  one  by  one  to  the 
model,  and  the  F-statistic  for  a  variable  to  be  added  must  be  the 
largest  among  the  outside  variables  and  it  must  be  at  the  f 
significance  level.  After  a  variable  is  added,  however,  the 
stepwise  procedure  looks  at  all  the  variables  already  included  in 
the  model  and  deletes  any  variable  that  does  not  produce  an  F- 
statistic  significant  at  the  b  significance  leval.  Only  after 
this  oheck  is  made  and  the  necessary  deletions  are  accomplished 
can  another  variable  be  added  to  the  model.  The  process  ends 
only  when  none  of  the  outside  variables  has  the  F  statistic 
significant  at  the  f  level,  and  none  of  the  inside  variables  has 
the  F  statistic  significant  at  the  b  level. 


Moreover,  the  value  of  R2  (the  coefficient  of  multiple 
determination)  is  also  taken  into  consideration,  when  the 
Stepwise  regression  analysis  is  performed.  The  R2  quantifies  the 
degree  of  association  that  exists  among  the  variables.  c  , 
Mallow's  statistic,  is  another  factor  which  is  used  in  generating 
a  stepwise  regression  model.  C.  was  introduced  by  Mallows  (1964) 
as  a  criterion  for  evaluating  the  adequacy  of  multiple  regression 
functions  of  differing  orders.  It  is  a  measure  of  total  squared 
error,  defined  as 


Cp  -  (  SSEp  /  s2  )  -  (  N  -  2*p  )  (82) 

where  s2  is  the  mean  square  error  (MSE)  for  the  full  model,  and 
SSE  is  the  sum-of-squares  error  for  a  model  with  p  parameters 
including  the  intercept,  if  any.  If  C  is  plotted  against  p, 
Mallows  recommends  the  modal  where  C  first  approaches  p.  When 
the  correct  modal  is  chosen,  the  parameter  estimates  are 
unbiased,  and  this  is  reflected  in  C  near  p.  If  the  full  model 
is  not  available  due  to  lack  ofp  a  "reasonable"  number  of 
observations,  only  the  value  of  R2  is  taken  into  the  decision¬ 
making  process. 


The  main  advantages  of  the  stepwise  procedure  are  that  it  is 
fast,  easy  to  compute,  and  availab] i  in  statistical  packages  for 
most  computers.  Also,,  the  cost  of  the  computation  rises  slowly 
as  the  number  of  variables  increases.  Another  advantage  of  the 
stepwise  procedure  is  that  it  possesses  order  independence, 
(i.e..  the  order  which  the  independent  variables  are  fed  into  the 
regression  model  does  not  affect  the  final  outcome  of  the 
analysis)  .  Unfortunately,  there  are  some  drawbacks  to  the  use  of 
stepwise  regression.  The  model  chosen  by  stepwise  regression 
need  not  be  the  best  by  criterion  of  interest;  indeed,  because  of 
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the  nature  of  the  one-at-a-time  philosophy  of  the  stepwise 
method,  there  is  no  guarantee  that  the  model  chosen  will  in  fact 
include  any  of  the  variables  that  would  be  in  the  best  subsot. 
The  stepwise  procedure  is  best  whan  the  independent  variables  are 
nearly  uncorrelated,  but  this  is  a  condition  that  in  seldom 
satisfied  in  practice. 


b.  Principal  Component  Analysis 


An  alternative  approach  to  alleviate  the  difficulty  of  having 
fewer  data  points  than  the  number  of  regressors  is  the  principal 
components  analysis  (PCA)  Method.  The  technique  uses  as  few 
principal  components  of  the  original  independent  variables  as 
possible  in  the  regression  model. 

Principal  components  P1f  P,,...,Pn  are  linear  combinations  of  the 
original  independent  variables,  say  x,,  x,,...,  x  (in  the  study's 
case  n«25) .  Pt,  P2,...,  F„  have  special  properties  in  terms  of 
variances;  for  example,  the  first  principal  component  P1  is  the 
normalized  linear  combination  with  maximum  variance.  The 
coefficients  in  the  linear  combinations  for  the  principal 
components  turn  out  to  be  the  normalized  eigenvectors  of  the 
variance-covariance  matrix  of  x^x,, . . .  ,xn,y,  called  £.  Since  £ 
is  a  non-negative  definite  matrix,  it  has  non-negative  real 
eigenvalues  a1  Z  A2  Z  ...  Z  An  and  associated  eigenvectors  a,, 

a2»  •  •  • i  an* 

Suppose  av  a2,...,  an  are  normalized;  i.e.,  a, 'a,  -  l,  where  ' 
means  "transpose."  Let  Z,  -  a,'X,  where  X  -  [xi;  x2,...,x]'  and  i 
■  l,2,...,n;  then  z,  is  the  ith  principal  component.  It  can  be 
shown  that  the  variance  of  Z,  is  the  eigenvalue  A.,,  where  i  - 
l,2,...,n.  The  idea  of  the  principal  components  method  is  to  use 
the  first  k  principal  components  as  the  independent  variables  in 
the  regression  model,  where  k  satisfies 

k 

—  z  C  }  where  C»  0.90,  0.95  (83) 

n 

£4‘ 


since  it  can  be  proven  that  the  first  k  principal  components  can 
explain  at  least  0*100%  of  the  total  variation  of  the  sampled 
standardized  software  failure  rates. 

Often,  k  is  much  less  than  n.  Therefore,  the  new  regression  model 
has  many  fewer  independent  variables  than  the  old  model,  but  it 
still  contains  all  the  original  characteristics,  since  these 
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independent  variables  are  linear  combinations  of  the  original 
variables. 

The  major  advantages  of  the  principal  components  method  are  that 
it  works  well  if  the  original  data  is  highly  correlated,  and  that 
all  the  information  pertinent  to  the  model  is  reserved.  Also,  as 
in  the  stepwise  procedure,  principal  components  analysis  is  order 
independent.  One  of  the  disadvantages  of  the  principal 
components  method  is  that  the  results  may  be  unstable  since  there 
is  too  much  data  contributing  to  the  model  (a  small  perturbation 
of  the  data  may  result  in  a  significant  change  in  the  result) . 
Another  drawback  of  this  technique  is  that  the  eigenvalues  and 
eigenvectors  of  Z  must  be  calculated,  but  Z  is  usually  large  and 
may  be  ill-conditioned;  therefore,  this  task  can  be  troublesome 
in  some  cases. 


4.2  Results 

4.2.1  Results  of  Prediction  of .Software  Failure  Rate 

Several  regression  models  were  formulated  to  describe  the 
relationship  between  the  software  failure  rate  and  the  software 
characteristics.  Comparisons  were  done  to  select  an  appropriate 
regression  model  to  be  used  to  predict  software  failure  rate  for 
each  phase  of  the  software  development. 

The  comparisons  were  done  based  on  several  statistical  values  of 
the  regression  models  (such  as:  R2,  p-values  of  hypothesis  tests 
(I)  and  (II),  standard  errors  of  the  parameter  estimates,  etc.), 
and  on  how  well  the  models  predict  known  software  failure  rates. 
In  addition  to  these  criteria,  the  following  ground  rules  were 
applied: 

1.  The  model  with  the  best  correlation  to  the  observed  software 
failure  rate  was  chosen  whenever  there  were  any  significant 
differences  in  correlation  between  the  two  models. 

2.  When  two  models  produced  essentially  identical  correlation 
between  the  predicted  software  failure  rate  and  the  failure 
rate  observed  on  the  projects,  the  model  which  accounted  for 
the  largest  number  of  metrics  (most  recent  information)  was 
selected. 

The  need  to  select  the  model  which  provided  a  software  failure 
rate  with  the  best  statistical  correlation  tc  the  observed 
failure  rate  is  obvious.  The  study  team's  preference  for  a 
prediction  model  which  used  the  largest  number  of  metrics  is 
based  on  two  areas  of  concern  with  respect  to  the  data  collected 
during  the  study  period.  The  data  collected  to  support 
development  and  validation  of  the  prediction  models  was  all 
collected  after  all  system  integration  testing  on  the  program  was 
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complete.  In  most  cases,  the  software  was  in  use  at  the  customers 
facility.  As  a  result,  the  metric  data  collected  on  tha  projects 
was  essentially  complete.  Thus,  the  data  available  to  the  study 
team  was  much  more  accurate  and  complete  than  the  data  which 
would  normally  be  available  at  the  end  of  a  given  program  phase 
(eg  requirements  analysis) .  This  may  have  resulted  in  muuh  better 
accuracy  for  the  early  program  phase  models  than  would  normally 
be  expected.  Also,  the  metric  data  collected  during  the  project 
was  limited  to  one  company  (Hughes  Aircraft)  and  thus  potentially 
valuable  variations  in  values  of  the  metric  data  which  would 
occur  between  companies  was  omitted,  possibly  yielding  better 
stability  and  accuracy  for  the  early  prediction  models  than  would 
ordinarily  result. 

A  complete  discussion  of  the  regression  analyses  is  provided  in 
Appendix  E.  Also,  details  of  the  comparisons  among  the  developed 
regression  models  are  included  in  the  Validation  section  of  this 
report. 


Three  regression  models  were  chosen  for  use  through  the  software 
development  phases  to  predict  software  failure  rate.  These 
models  are: 

-  Modal,  M2, l 

Y  -  18.04  +  0.05  P,  (Y  is  restricted  to  be  non-negative) 

P,  -  0.009X,  +  0.99X2  +  0. 10Xj  -  0.0001x4  4-  0.0005xs 

-  Model  Mfi.1 

Y  »  17.90  +  0.04  P1  (Y  is  restricted  to  be  non-negative) 

P,  -  0.007X,  +  0.796x2  +  O.O8X3  -  0.0Q03X4  4-  0 , 0003Xj  4- 
0.00009xfl  4-  0.0043X7  4-  0.013xB  4-  0.6X,  4-  O.OO3x10 

-  Model  M10: 

Y  ■  17.88  +  0.04  P,  (Y  is  restricted  to  be  non-negative) 

PA  *  0 . 007X1  +  0.80x2  4-  0 . 08Xj  +  0.01xg  +  0 . 6x?  +  O.OOSx23  + 


M3  is  used  to  predict  the  software  failure  rate  during  the 
Preliminary  Design  phase.  M6  can  then  be  used  during  the 
Detailed  Design  phase  and  the  Code  and  Unit  Test  phase.  Model 
M10  ran  be  used  after  Code  and  Unit  Test  phase  has  passed. 
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The  relationship  between  software  fault  content  and  the  software 
failure  rate  is  as  follows:  Let  0,  be  the  hazard  rate  of  the  ith 
software  fault  in  a  program.  Then  the  overall  program 
failure  rate  is 


o 


(84) 


The  Basic  Execution  Time  Model,  discussed  more  fully  in  Section  5 
of  this  report,  asserts  that  every  fault  hazard  rate  0,  is  equal 
to  a  constant,  the  per- fault  hazard  rate  <p.  In  this  case  the 
overall  program  failure  rate  is 


■s 


(85) 


The  multiple  regression  models  predict  a  standardized  failure 
rate  that  corresponds  to  the  Basic  Execution  Time  Model's  X0 
parameter.  To  obtain  the  same  two  Basic  Execution  Time  Model 
parameters  that  are  supplied  by  estimation  during  growth  testing 
(see  Section  5),  v0  and  j9,  the  following  formulas  are  employed. 
For  total  failures, 


f  •  K‘B 


(86) 


and  for  0, 


P  *  B  •  *  B  •  f  •  K 


(87) 


where  the  linear  execution  frequency  f  is 

f  *  -j  (88) 

The  linear  execution  frequency  is  the  number  of  program  passages 
there  would  be  in  unit  execution  time  if  the  program's 
instructions  were  executed  in  linear  sequence.  I  is  the  number 
of  object  instructions  and  is  obtained  by  multiplying  the  number 
of  source  instructions  by  a  code  expansion  ratio.  Table  4-2 
provides  the  code  expansion  ratios  for  selected  programming 
languages  [Jones  (1986)]. 
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Table  4-2.  Code  Expansion  Ratios 


Programming  Language 

Expansion  Ratio 

Assembler 

1 

Macro  Assembler 

1.5 

C 

2.5 

COBOL 

3 

FORTRAN 

3 

JOVIAL 

3 

Ada 

4.5 

A  method  of  relating  fault  content  (obtained  from  applicable 
product/process  related  metrics)  to  failure  rate,  based  on 
program  structure  and  operational  profile  was  developed.  In  the 
framework  of  the  Basic  Execution  Time  Model,  fault  content  Is 
related  to  failure  rate  through  the  fault  exposure  ratio,  K. 
Specifically,  if  there  are  u0  faults  in  the  program,  then  the 
failure  rate  is 

10  -  0  (89) 


With  regard  to  faults,  the  failure  rate  depends  primarily  on 
fault  density,  not  on  fault  content.  The  initial  failure  rate, 
A0,  and  the  inherent  faults,  <o0,  are  the  failure  rate  and  fault 
content  at  the  start  of  system  test,  but  ih  actuality  any  point 
during  system  test  could  be  arbitrarily  designated  the  "official 
start"  of  system  test,  so  the  relationship  holds  throughout 
system  test  and  during  field  use. 

Zeroing  in  on  a  single,  "typical"  fault,  the  Basic  Execution  Time 
Model  defines  the  per-fault  hazard  rate  as 

<p  -  /  •  K  (90) 


The  per-fault  hazard  rate  is  the  equal  contribution  each  fault 
makes  to  the  overall  program  failure  rate.  The  method  developed 
also  handles  the  situation  in  which  a  program  departs  from  that 
assumption.  In  that  situation,  faults  will  fall  into  various 
hazard  rate  classifications  (hazard  rata  profile)  (SOW  para. 
4. 1.3. 3.1) . 

Musa  introduced  the  factor  K  to  account  for  two  distinct 
phenomena,  the  dynamic  structure  of  the  program  and  the  varying 
machine  state.  The  dynamic  structure  of  the  program  is  the 
looping  and  conditional  and  unconditional  branching  that  takes 


place  during  execution.  The  machine  state  ie  the  values  of  all 
program  variables.  The  machine  state  can  change  upon  execution 
of  each  instruction.  When  a  fault  is  encountered  during 
execution,  a  failure  may  or  may  not  occur  during  a  particular 
encounter;  it  depends  on  the  machine  state.  When  K-l, 

<t>  -  f  (91) 

This  occurs  only  when  the  program  is  executed  linearly  and  all 
faults  cause  failures  regardless  of  the  machine  state. 

The  study  team  separated  K  into  two  factors 

K  -  8  'M  (92) 

where  S  is  a  structure  factor  and  H  is  a  machine  state  factor. 
In  relation  to  a  single  fault,  S  and  K  are  defined  as 

Mm  failure  "causing  machine  states 
total  machine  states 


actual  encounters 
linear  encounters 


Both  M  and  s  are  multiplicative  factors  because  doubling  either 
of  them  doubles  the  failure  rate.  So  now 

4>  •  f  '  3  M  (95) 


The  product  f  •  S  is  the  frequency  with  which  a  fault  is 
encountered.  Once  the  fault  is  encountered,  the  probability  of  a 
failure  ensuing  is  M.  Stochastically,  S  is  the  probability  of  an 
arbitrary  fault  being  encountered  during  one  program  passage  (of 
which  there  are  f  per  unit  time) .  M  is  the  probability  of  a 
failure  ensuing  (because  of  an  unfavorable  machine  stato) ,  given 
that  a  fault  is  encountered. 

If,  as  in  the  Basic  Execution  Time  Model,  K  is  assumed  to  be 
constant  for  a  particular  program,  what  about  S  and  M?  Since  K  - 
S  •  M  is  a  constant,  there  are  two  possibilities;  (1)  s  and  M 
are  themselves  constant;  or  (2)  S  and  M  countervail  each  other  so 
as  always  to  equal  a  constant.  Alternative  2  is  the  suggestion 
that  when  S  is  large,  M-K/S  is  small,  and  when  M  is  large  S-K/M 
is  small. 

Since  S  is  a  function  of  the  structure  (loops,  branches)  of  the 
program,  and  M  is  a  function  of  the  arithmetic  and  logic  within 
the  fault,  it  is  reasonable  to  assume  that  S  and  M  are 
independent  quantities.  Hence,  since  K-S- M  is  a  constant,  and  S 
and  M  vary  independently,  it  can  be  concluded  that  S  and  M  are 
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each  by  itself  a  constant  for  a  particular  program,  which  is 
alternative  1.  The  ramification  of  K  being  separable  into  S  and 
M  is  that  the  two  quantities  can  be  studied  individually.  If  K 
is  affected  by  the  structure  of  the  program,  then  the  effect 
should  be  seen  in  S  and  not  M. 

S  is  the  rate  at  which  a  particular  fault  is  encountered  during  a 
single  program  passage.  One  way  to  estimate  S  would  be  to  inject 
a  program  with  a  single  fault  and  see  how  often  that  fault  is 
encountered.  By  injecting  several  faults,  a  more  accurate  and 
quicker  estimate  would  be  obtained.  But  as  discussed  earlier  in 
this  section,  it  is  difficult  to  fabricate  faults  that  are 
representative  of  a  program's  indigenous  faults. 

But  why  inject  a  fault?  A  fault  is  in  essence  just  a  place  in 
the  program.  The  major  issue  is  how  often  an  arbitrary  place  in 
the  program  is  reached  per  program  passage.  S  can  be  estimated  by 
observing  how  often  one  or  more  randomly  chosen  places  in  a 
program  are  encountered  during  execution.  The  assumption  that  is 
being  made  here  is  that  a  fault  can  occur  anywhere  in  a  program, 
and  every  place  in  the  program  is  as  likely  as  any  other  to  have 
a  fault.  That  is,  the  fault  density  of  a  program 
(faults/kilolines  of  executable  code)  is  uniform.  This 
assumption  has  been  borne  out  through  empirical  studies  [Basili 
and  Hutchens  (1983),  Takahashi  and  Kamayachi  (1985)]. 

Here,  then,  is  the  method  developed  for  estimating  S  and  using 
that  estimate  to  refine  K  to  better  reflect  program  structure  and 
operational  profile. 

1.  An  array  of  counters  is  declared.  In  Ada,  the  declaration 
would  be 

counter:  array  (l..n)  of  integer  :  =  (0 , 0 , 0 0)  ;  (96) 


2.  A  random  sample  is  taken  of  n  "places"  within  the  program. 
Thase  places  must  lie  in  between  existing  program  statements.  At 
each  place  a  counter  iteration  statement  is  inserted.  For 
example,  the  following  statement  is  inserted  into  the  program  at 
the  i-th  such  place: 

counter (i)  counter  (i)  +  1;  (97) 


3.  The  program  is  executed  for  w  time  units.  During  the  period 
of  execution  the  operational  profile  should,  ideally,  be 
stationary  and  representative  of  field  use.  When  the  execution 


period  is  complete,  the  values  of  elements  of  the  COUNTER  array 
are  examined,  and  the  sample  values  for  S  are  calculated  from 


COUNTER  U) 
f  *  w 


i=l,  2, ...,  n 


(98) 


Note  that  f*w  is  the  number  of  program  passages  that  take  place 
during  the  w  time  units.  If  S  is  a  constant  for  a  particular 
program,  as  implied  by  the  Basic  Execution  Time  Model,  then  any 
variability  in  the  observed  values  {s,}  can  be  attributed  to 
"sampling  fluctuations."  The  best  estimator  for  s  will  be  the 
sample  mean 


5 


(99) 


Under  the  assumption  that  K  is  constant,  the  Gaussian  law  of 
errors  applies.  The  sample  standard  deviation  is 


a 


(si-S)*/(fl“l) 


(100) 


A  100(l-a)%  confidence  interval  for  S  has  endpoints 

S  *  tn-x.i-./a-—  (101) 

where  tn-1  ,.a/2  is  that  value  of  the  t-distribution  with  n-1 
degrees  o'f  freedom  that  has  l-a/2  to  the  left.  The  t- 
distribution  is  tabulated  in  many  statistics  textbooks.  A 
numerical  method  suitable  for  programmable  calculators  and 
computers  is  given  in  [Volk  (1982)].  For  n>30,  tn.,  ,.^2  can  be 
replaced  by  the  standard  normal  deviate  kv<|/2. 

If  no  information  is  available  about  K  for  this  program,  the 
industry  average  value  of  4.20  x  10’7  is  recommended  [Musa  et  al. 
(1987)].  If  the  same  or  a  similar  program  has  gone  through 
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growth  testing,  then  the  value  can  be  obtained  from  the  following 
parametric  relationship: 


K  * 


1 

Bf 


(102) 


In  any  case,  the  value  of  M  can  then  be  recovered  as 

■  -f  (103) 


Here  is  a  summary  of  the  dimensions  of  the  various  quantities 
involved  in  predicting  the  parameters  of  the  Basic  Execution  Time 
Model : 


f,  program  passages 
execution  time 


(104) 


Si 


encounters 


faults 


program  passages 


Mi 


Ki 


failures 

encounters 


failures 


faults 


program  passages 


In  • 


(>)g:  faults 


failures 


execution  time . 


(105) 

(106) 

(107) 

(108) 

(108) 


If  the  s.  values  are  highly  variable  or  skewed,  then  the 
assumption  of  a  constant  per-fault  hazard  rate  should  be 
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questioned.  Skewness — the  degree  of  symmetry  in  the  shape  of  the 
distribution — can  be  measured  by 

Sk  -  3  (l~~  MdL  (110) 

8 

where  Md  is  the  median  of  the  {s,}.  A  large  positive  score 
indicates  that  the  distribution  is  skewed  to  the  right,  and  a 
large  negative  score  indicates  that  it  is  skewed  to  the  left. 
For  the  purposes  of  prediction,  a  skewed  distribution  suggests 
that  the  Basic  Execution  Time  Model  may  not  be  appropriate. 

If  the  {s,}  are  highly  skewed  or  highly  variable,  then  the  hazard 
rate  for  each  counter  should  be  computed  individually  as 

■  Sj  •  Jf  •  f  (ill) 

The  distribution  of  hazard  rates  can  be  expressed  through  a 
"hazard  rate  profile."  The  range  of  fault  hazard  rate  values  is 
partitioned  into  n  subranges,  which  will  be  termed  classes. 
Adams  (1984)  researched  a  similar  situation  and  devised  a 
classification  scheme.  Adapted  here  to  the  purposes  of  fault 
hazard  rate  classification,  the  first  step  is  to  find  the  highest 
fault  hazard  rate,  <£hlgh,  and  the  lowest  fault  hazard  rate,  <f>{  . 
Starting  from  0h)  h,  the  class  boundaries  are  formed  by 
successively  multiplying  0hJgh  by  10*^0.31623: 

Qhljth'  *  $hl0h  ( 10"  a*  ( lO"'*)  J  (112) 


Denote  the  proportion  of  fault  hazard  rates  falling  into  the 
classes  as 


Pi'  Pi'  "• '  Pn 


(113) 


with 


(114) 


Let  the  mean  hazard  rate  value  in  each  class  j  be  denoted  m, . 
Class  j  as  a  whole  contributes  o0p{m,  to  the  overall  program 
failure  rate,  which  will  be 

*o  ■  Mo U15) 


If  a  similar  program  is  predicted  to  have  a  fault  content  of  u0, 
and  it  is  presumed  that  the  hazard  rate  profile  is  the 
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(116) 


same,  than  the  program  failure  rate  is 


4.2.3  Eaulfc.  Rtdustlgn  Factor 

Another  quantity  that  appears  in  the  Basic  Execution  Time  Model 
is  the  fault  reduction  factor,  B.  The  fault  reduction  factor  is 
the  net  number  of  faults  removed  per  failure  occurrence.  B  is 
not  generally  equal  to  unity  because  of  Imperfect  debugging. 
Repair  activity  can  fail  to  find  the  causative  fault,  can  find 
and  remove  several  related  faults,  or  can  inadvertently  generate 
new  faults.  The  fault  reduction  factor  can  be  computed  [Musa 
(1984)]  from 

B  »  D{  1+A)  (1-0)  (117) 

where  D  is  the  "detectability  ratio"— the  proportion  of  faults 
whose  causative  failures  can  be  found.  The  quantity  A  is  the 
"associability  ratio"— the  ratio  of  faults  discovered  by  code 
reading  to  faults  discovered  by  testing.  G  is  the  "fault  growth 
ratio"— the  increase  in  faults  per  fault  corrected.  Generally,  D 
is  close  to  1,  A  is  close  to  0,  and  G  ranges  from  0  to  0.91 
[Miyamoto  (1975),  Musa  (1980)]. 

If  data  from  growth  testing  on  a  similar  project  is  available, 
then  B  can  be  obtained  from  the  relationship 

B  9  ,  <118> 


where  0  is  a  parameter  of  the  Basic  Execution  Time  Model  (see 
Section  5)  . 

Another  option,  when  no  information  is  available,  is  to  use  the 
industry  average  figure  0.955  [Musa  et  al.  (1987)]. 

4.2.4  Summary  of  Prediction  Technique 


a.  With  no  prior  information,  use 

w  M  _ 6  faults _ 

1  1000  source  instructions 


(119) 


Predict  the  number  of  developed  source  instructions  Als  (omitting 
reused  code) .  Then  compute  the  inherent  fault  density  as 

«0  -  «r  *  (120) 
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Use  0.955  for  B,  or  obtain  it  from  historical  data,  and  obtain 
the  total  failures  as 


vo  * 


(121) 


Using  the  appropriate  code  expansion  ratio,  compute  the  number  of 
object  instructions  I.  From  manufacturer's  specifications  or 
benchmarking,  find  out  the  average  instruction  execution  rate  r. 
Obtain  the  linear  execution  frequency  as 

/  ■  -j  (122) 


Use  4.20  x  io‘7  as  the  value  for  K,  or  obtain  X  from  historical 
information  on  a  similar  project.  Predict  the  initial  program 
failure  rate  as 

X0  ■  (123) 


and  the  time-dependent  failure  rate  as 


l(t) 


l0exp 


T 


(124) 


where  r  is  cumulative  execution  time  since  the  start  of  system 
test. 


b.  As  product/process  metrics  come  in  during  requirements 
analysis,  preliminary  design,  detailed  design,  and  coding  and 
unit  test,  use  the  regression  model  to  obtain  a  better  prediction 
of  10. 

c.  Once  system  test  begins,  use  actual  failure  data  to  estimate 
the  software  reliability  model  parameters,  as  described  in 
Section  5. 


so 


Reliability  growth  is  the  positive  improvement  in  reliability 
over  time.  For  hardware,  reliability  growth  occurs  over  a  period 
of  time  due  to  improvements  in  the  product  design  or 
manufacturing  process.  For  software,  reliability  growth  occurs 
through  systematic  testing  and  debugging  resulting  in  effective 
removal  of  software  faults. 

Growth  management  is  part  of  the  system  engineering  process,  and 
complements  bther  basic  reliability  program  activities  such  as 
prediction,  allocation,  and  demonstration  testing.  Growth 
management  consists  of  planning  the  achievement  of  reliability  as 
a  function  of  time  and  other  resources. 

The  rate  of  software  reliability  growth  depends  on  how  fast  the 
test-debug-test  loop  can  be  accomplished.  The  rate  is 
constrained  by  the  failure  rate  of  the  software,  the  amount  of 
computer  time  available,  and  the  availability  of  failure 
identification  personnel  (testers)  and  failure  resolution 
personnel  (debuggers) .  The  rate  is  controlled  by  reallocation  of 
resources  and  other  forms  of  intervention,  based  on  comparisons 
between  planned  and  assessed  reliability  values. 


In  fulfillment  of  SOW  Task  4,  a  standard  methodology  for 
reliability  growth  testing,  measurement,  and  management  was 
developed.  The  techniques  are  built  around  a  time  domain 
software  reliability  model.  Methods  were  developed  for  selecting 
random  test  cases  from  an  operational  profile  that  emulates  field 
use,  and  established  procedures  for  data  collection,  parameter 
estimation,  determination  of  the  calendar  time/execution  time 
ratio,  assessment  of  model  goodness-of-fit,  recalibration,  and 
use  of  multiple  copies. 

5.1  Abb coach 

First,  criteria  were  identified  for  selecting  an  appropriate  time 
domain  software  reliability  model  around  which  to  build  the 
growth,  testing  methodology  (SOW  para.  4. 1.4.1).  An  important 
consideration  was  that  the  model  had  to  be  compatible  with 
existing  hardware  reliability  standards,  concepts,  and 
procedures.  The  model  should  be  mathematically  tractable,  so 
that  the  formulas  can  be  easily  invertible  to  find  out  various 
quantities  of  interest  (such  as  forecasting  when  a  future 
reliability  objective  will  be  met).  The  parameters  of  the  model 
needed  to  have  simple  interpretations  that  are  meaningful  to 
software  engineers. 

Another  important  consideration  is  the  predictive  validity  of  the 
model.  Ideally,  the  model  should  be  one  that  has  been  validated 
on  a  broad  selection  of  software  development  projects.  Because 
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no  software  reliability  model  has  been  completely  validated  on 
all  types  of  applications  across  different  organizations,  it  was 
decided  that  the  growth  testing  methodology  should  include  a 
goodness-of-fit  technique  and  a  recalibration  technique.  A 
goodness-of-f it  technique  would  allow  a  user  to  gauge  how  well 
the  software  reliability  model  is  working  on  a  given  program.  If 
the  fit  is  poor,  the  user  can  switch  to  an  alternative  model  or 
parameter  estimation  technique.  In  many  cases,  the  reason  for 
poor  fit  will  turn  out  to  be  a  systematic  bias  (consistent 
optimism  or  pessimism)  or  noisiness  in  the  estimations.  These 
problems  can  be  corrected  by  recalibration.  The  study  team 
sought  to  develop  a  new  type  of  recalibration  technique  that  is 
numerical,  as  opposed  to  the  existing  graphical  techniques.  A 
numerical  technique  is  more  accurate  and  has  the  advantage  of 
being  easily  implemented  on  a  programmable  calculator  or 
computer . 

The  team  paid  special  attention  to  the  procedures  for  conducting 
growth  testing.  The  prevailing  objective  of  testing  in  recent 
years  has  been  to  aggressively  seek  faults  by  testing  at  high 
workload  and  at  boundary  values  of  the  input  variables.  This 
type  of  testing  should  be  completed  prior  to  growth  testing. 
During  growth  testing,  the  input  states  must  be  chosen 
"statistically,"  which  is  to  say  that  the  input  states  must  be 
randomly  selected  from  a  stationary  operational  profile 
(probabilities  of  input  states) .  The  operational  profile  during 
growth  testing  should  be  close  to  the  operational  profile  the 
program  will  experience  during  field  use.  Statistical  testing  is 
effective  in  contributing  to  improved  reliability,  because  it 
finds  the  faults  that  the  user  will  tend  to  encounter  most.  On 
the  other  hand,  stress  testing  or  coverage  testing  might  find 
many  faults,  but  because  the  user  will  rarely  encounter  those 
faults,  their  removal  may  not  significantly  improve  the  failure 
rate. 


5.2  BfigUl&a 

The  team  chose  a  variation  of  the  Basic  Execution  Time  Model 
[Musa  et  al .  (1987)]  as  the  time  domain  software  reliability 
model  on  which  to  base  the  growth  testing  methodology.  This 
model  has  a  wide  following  among  software  practitioners  in  large 
corporations.  Much  practical  knowledge  and  many  auxiliary 
techniques  have  come  out  of  the  model's  real-world  experience, 
which  enable  the  model  to  be  adapted  to  a  wide  variety  of 
software  development  technologies  and  project  situations. 

The  Basic  Execution  Time  Model  has  simple  parameters  that  are 
easy  for  software  engineers  and  project  managers  to  understand 
and  to  which  they  can  relate.  The  model  is  fully  compatible  with 
existing  hardware  reliability  assessment  methods  and  is 
mathematically  tractable.  The  data  needed  to  estimate  the  model 
parameters  is  minimal  and  relatively  easy  to  collect.  Another 
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reason  for  choosing  the  Basic  Execution  Time  model  is  that  it 
represents  a  synthesis  of  many  of  the  advances  that  have  taken 
place  in  almost  20  years  of  research  in  the  field  of  software 
reliability  modeling. 

5.2.1  Major  Time  Domain  Software  Reliability  Models 

Jelinski  and  Moranda  (1972)  developed  the  first  reliability 
growth  model  created  specifically  for  software.  The  model 
assumes  that  the  failure  rate  is  linearly  proportional  to  the 
number  of  faults  in  the  software.  When  a  failure  occurs  it  is 
assumes  that  the  fault  that  caused  it  is  removed  instantaneously, 
without  spawning  any  new  faults.  The  failure  rate  remains 
piecewise  constant  between  failures,  and  drops  by  a  constant 
amount  at  each  failure.  The  interfailure  times  are  independent 
exponentially  distributed  random  variables.  Around  the  same 
time,  Shooman  (1973)  presented  a  similar  model  but  introduced 
additional  concepts:  The  failure  rate  is  not  only  a  function  of 
the  number  of  faults  in  the  program,  it  is  also  a  function  of  the 
instruction  processing  rate,  the  program  size,  and  the  program 
structure.  Fault  correction  occurs  at  a  different  time  than 
failure  occurrence  and  is  affected  by  the  nature  of  the  project 
and  the  number  of  available  personnel. 

Musa  (1975)  developed  a  modal  that  built  on  the  Jelinski-Moranda 
and  Shooman  models  and  included  several  new  concepts.  Execution 
(CPU)  time,  not  calendar  time,  is  the  failure-inducing  stress 
placed  on  a  program.  During  field  operation  the  failure  rate 
remains  constant;  only  during  .periods  of  fault  correction 
activity  does  the  failure  rate  change.  Musa  accounts  for  the 
dissimilarity  between  the  testing  and  operational  environments  by 
means  of  a  testing  compression  factor. 

Goel  and  Okumoto  (1979)  presented  a  model  based  on  a 
nonhomogeneous  Poisson  process  (NHPP) .  This  model  assumed  that 
the  number  of  faults  in  the  program  at  the  start  of  testing  is 
not  fixed  but  is  a  Poisson  random  variable.  In  earlier  models 
each  interfailure  period  is  governed  by  a  distinct  (homogeneous) 
Poisson  process,  whereas  now  the  entire  system  test  phase  is 
governed  by  a  single  nonhomogeneous  Poisson  process.  At  the 
expense  of  conformance  to  an  intuitive  desire  for  a  piecewise 
continuous  failure  rata,  Goel  and  Okumoto  achieve  the  analytical 
simplicity  of  a  smooth  failure  rate  curve.  New  concepts  included 
the  idea  that:  repair  of  the  underlying  fault  occurs  at  some 
variable  time  after  failure  occurrence.  Other  concepts  included 
were  that  a  program  can  be  "perfect"  and  have  no  faults  and  that 
the  effectiveness  of  repair  actions  is  imperfect. 

The  concept  of  imperfect  debugging  is  an  important  one.  Evidence 
strongly  suggests  that  most  fuults  originate  in  the  requirements 
and  design  phases  [Lipow  (1979)].  A  fault  caused  by  a 
requirements  misunderstanding  or  design  deficiency  may  require 
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extensive  reworking  of  the  program,  and  this  reiteration  of  the 
original  development  process  will  likely  spawn  new  faults. 

Musa  and  Okumoto  created  a  model  called  the  Basic  Execution  Time 
Modal  [Miim  at  al .  (1987)]  that  embodies  all  of  the  foregoing 

concepts . 

One  concept  championed  by  Littlewood  (1978)  was  that  different 
faults  in  the  program  make  different  contributions  to  the  overall 
failure  rate.  The  most  frequently  encountered  faults  (those  that 
contribute  most  to  the  overall  program  failure  rate)  are  detected 
and  corrected  first.  Because  some  programs  are  subjected  to  a 
highly  nonuniforw  operation  profile,  Musa  and  Okumoto  developed  a 
model,  called  the  Logarithmic  Poisson  Model,  that  considers  the 
hazard  rates  of  the  faults  in  the  program  to  form  a  geometric 
progression,  an  idea  similar  to  Littlewood’s.  The  Logarithmic 
Poisson  Model  should  be  tried  if  the  goodness-of-fit  procedure  is 
showing  a  poor  fit  with  the  Basic  Execution  Time  Modal. 

The  Basic  Execution  Time  Model  (as  well  as  the  Logarithmic 
Poisson  Model)  is  a  nonhomogeneous  Poisson  process  (NHPP)  model. 
The  NHPP  framework  is  tho  most  commonly  used  for  modeling 
reliability  growth.  An  NHPP  conforms  to  four  axioms.  The 
failure  counting  process  (N(r),r2*0),  which  gives  the  cumulative 
number  of  failures  in  the  execution  time  interval  [  p ,  r  ] ,  is  an 
NHPP  with  intensity  function  (time-dependent  failure  rate)  l(r), 
if 

.  V  \ 

(i)  N(0)  -  o?  1 

(ii)  (N(r),TbO)  has  independent  increments; 

(ill)  Pr{N(r+AT)-N(T)£2)  -  o(Ar) ; 

(iv)  Pr  (N(t+At) -N(t)«1)  -  X(r) At+o(At)  > 

where  the  function  o(At)  is  defined  by 


lim-2JAlL  .  o  ( 12S) 

Ai-0  At 


The  mean  value  function  of  the  NHPP  is  the  expected  number  of 
failures  occurring  in  the  execution  time  interval  [0,r]  and  is 
defined  by 


H  (t)  ■  ( s)  ds  (126) 

o 
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Yamada  and  Osaki  (1983)  surveyed  the  NHPP  applied  to  growth 
modeling.  One  of  the  most  prominent  models  is  the  power  law 
model  or  the  Duane  model.  It  was  developed  empirically  by  Duane 
(1964)  and  is  often  called  the  Duane  model.  Duane  based  his 
model  on  the  observation  that  the.  plot  of  cumulative  failure  rate 
versus  cumulative  test  time  closely  follows  a  straight  line  on 
log-log  paper.  Crow  (1974)  represented  his  postulate 
stochastically  as  an  NHPP  with  a  Weibull  failure  rate  curve  and  a 
mean  value  function  of 


( t)  -  A,  t( 


(127) 


where  (  it  a  growth  rate  parameter,  generally  between  0.1  and 
0.65.  This  model  is  also  known  as  the  AMSAA  model. 

Since  the  Basic  Execution  Time  Model,  which  has  been  selected  is 
also  an  NHPP  model,  the  failure  rate  projections  of  the  AMSAA  and 
Basic  Execution  Time  models  can  be  combined  to  project  the  system 
reliability  of  concurrent-sequential  hardware/software  systems. 

5.2.2  Selected  Software  Growth  Model 

5. 2. 2.1  Failure  Component 

A  software  failure  is  a  departure  in  program  output  (hardcopies, 
displays,  commands,  control,  etc.)  from  that  specified  by  the 
requirements.  The  "departure"  from  requirements  consists  of  an 
output  variable  that  differs  from  its  correct  value.  Each 
incorrect  output  variable  is  counted  as  a  software  failure  in  its 
own  right  if  it  arose  from  a  separate  fault.  This  definition  of 
software  failure  differs  slightly  from  that  used  in  demonstration 
testing,  where  the  simultaneous  appearance  of  several  incorrect 
output  variables  would  count  as  a  single  failure.  In  growth 
testing,  if  several  output  variables  have  incorrect  values  and 
the  discrepancies  were  caused  by  multiple  faults,  then  fault 
correction  activity  will  address  all  those  faults  and  the  failure 
rate  will  improve  by  several  decrements,  just  as  if  the 
discrepancies  had  occurred  over  the  course  of  several  runs. 

As  a  program  is  debugged,  the  amount  of  execution  time  that 
elapses  between  failures  tends  to  increase.  In  the  Basic 
Execution  Time  Model,  it  is  postulated  that  each  failure  results 
in  fault  correction  activity  that  causes  the  failure  rate  to  drop 
by  a  fixed  amount.  When  the  program  is  released,  the  code  is 
frozen  and,  as  long  as  the  operational  profile  is  stationary,  the 
software  will  exhibit  a  constant  failure  rate  (see  Appendix  B) . 

Let  v0  be  the  (expected)  total  failures.  This  is  the  number  of 
failures  that  need  to  occur  to  expose  and  remove  all  faults. 
This  parameter  can  be  predicted  (see  Section  4)  or  can  be 
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estimated  from  observed  failure  data.  Exposing  and  removing  all 
faults  can  conceivably  take  an  infinite  amount  of  time.  Let  A0 
be  the  initial  failure  rate,  at  the  start  of  system  test  (r»0). 
This  parameter  can  also  be  predicted  (see  Section  4)  or  can  be 
estimated  from  observed  failure  data.  Any  stress  testing  should 
take  place  prior  to  r-0.  If  ju  is  the  mean  cumulative  number  of 
failures  experienced  at  some  point  after  the  start  of  system 
test,  the  failure  rate  as  a  function  of  /i  is 


A  (^)  ■  A0  (l-^r 

\  vo/ 


(128) 


Thus  the  failure  rate  k(n)  is  a  linear  function  of  mean  failures 
experienced  m.  Note  that  the  failure  rate  A.  (At)  is  a  smooth 
function.  In  early  software  reliability  models,  the  failure  rate 
drops  disoontinuously  at  each  failure;  in  the  Basic  Execution 
Model  the  failure  rate  glides  smoothly  down  to  reflect  the 
uncertainty  in  the  times  that  the  faults  are  actually  removed. 
The  slope  of  failure  rate  is 


d\ 

4 1  '  v0 


(129) 


The  decrement  of  failure  rate  per  failure  as 

Another  way  to  express  the  failure  rate  is  as  the  product  of  0 
and  the  number  of  failures  remaining: 

l(n)  ■  (J (v0-h)  (130) 

or,  since  the  mean  failures  experienced  is  a  function  of 
execution  time  r : 


A,  (x)  ■  P  [v0-n(t)] 


(131) 


The  function  h(t)  is  called  the  mean  value  function.  The  time 
derivative  of  the  mean  value  function  is  ths  time-dependent 
failure  rate  A(r).  Thus  one  can  form  the  differential  equation 
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■d^J;  )  ♦  P|A(t)  =  pv0 


(132) 


the  solution  to  which  is 

H(t)  ■  v0(l-exp[-pt] )  (133) 

Differentiating,  ona  obtains  the  failure  rate 

Mt)  «  v0Pexp[-pt]  (134) 

The  initial  failure  rate  is  found  by  setting  r-0: 

X0-V0p  (13S) 


The  model  is  thus  completely  determined  by  two  parameters:  0  and 
the  total  failures  v0. 


5. 2. 2. 2  Fault  Component 

A  software  fault  is  an  extra,  missing,  or  defective  instruction 
or  set  of  instructions  that  has  caused,  or  can  potentially  cause, 
a  software  failure.  When  a  fault  is  encountered  during 
execution,  a  software  failure  may  or  may  not  ensue,  depending  on 
the  machine  state  (values  of  program  variables) .  The  growth  that 
is  being  modeled  is  in  the  reliability  (decrease  in  failure 
rate) .  The  number  and  location  of  faults  affects  the  reliability 
only  indirectly. 

The  model  contains  additional  elements  to  describe  the  role  of 
software  faults.  The  number  of  failures  experisnced  and  the 
number  of  faults  removed  are  related  by  a  quantity  called  the 
fault  reduction  factor,  denoted  B.  The  fault  reduction  factor  is 
the  net  number  of  faults  removed  per  failure.  B  accounts  for 
imperfect  debugging.  Ordinarily,  the  causative  fault  behind  a 
failure  is  found  and  removed.  But  sometimes  the  causative  fault 
will  not  be  found,  resulting  in  zero  faults  removed.  Other 
times,  the  failure  will  result  in  the  discovery — by  code  reading- 
-and  removal  of  several  related  faults.  Sometimes  fault 
correction  activity  will  introduce  additional  faults  into  the 
program.  The  average  number  of  faults  removed  per  failure  is  B. 
Since  the  occurrence  of  one  failure  results  in  the  failure  rate 
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declining  by  the  quantity  0,  the  amount  that  each  fault 
contributes  to  the  overall  program  failure  rate  must  be 


( p  a 


2 

B 


U3«> 


which  will  be  termed  the  per-fault  hazard  rate.  It  is  called  a 
hazard  rate  (force  of  mortality  [FOM])  because  a  fault  has  a 
lifetime.  When  the  fault  is  removed  it  is  gone  for  good.  The 
overall  program  has  a  failure  rate  (rate  of  occurrence  of 
failures  [ROCOF])  because  it  is  "repairable"  via  restart. 

Let  the  number  of  inherent  faults  be  denoted  o0.  This  is  the 
number  of  faults  present  at  the  start  of  system  test  and  does  not 
include  any  faults  introduced  later  as  the  inadvertent  result  of 
repair  activity.  The  initial  failure  rate  can  then  be  expressed 
as  the  product  of  the  per-fault  hazard  rate  and  the  number  of 
inherent  faults. 


■  0uo 


(137) 


The  per-fault  hazard  rate  $  can  be  further  broken  apart  into 


4>  -  /  •  K  (133) 

The  quantity  K  is  called  the  fault  exposure  ratio  and  is 
discussed  below.  The  quantity  f  is  the  linear  execution 
frequency  of  the  program,  l.e.,  how  many  times  the  program  would 
be  executed  per  unit  time  if  its  instructions  were  executed  in  a 
linear  sequence.  The  linear  execution  frequency  can  be 
calculated  by  dividing  the  average  instruction  execution  rate  r 
by  the  number  of  machine-level  instructions  I: 

f  »  j  (13*) 

The  average  instruction  execution  rata  is  the  number  of 
instructions  executed  per  unit  time.  It  is  frequently  expressed 
in  terms  of  MIPS  (millions  of  instructions  per  second) .  The 
"MIPS  rating"  of  a  computer  is  often  available  from  the  vendor 
and  can  be  refined  by  customer  benchmarking  to  reflect  the 
instruction  mix  of  the  customer's  own  applications.  To  get  the 
number  of  machine-level  instructions,  it  is  easiest  to  multiply 
the  number  of  source-level  instructions  I,  by  an  average 
expansion  ratio  for  the  programming  language,  such  as  tabulated 
by  Jones  (1986) . 
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In  general,  programs  are  not  executed  linearly.  They  contain 
looping  constructs  and  unconditional  and  conditional  branches,  so 
the  number  of  times  an  individual  fault  is  encountered  per  unit 
time  can  be  greater  or  less  than  the  linear  execution  frequency 
f.  Every  time  a  fault  is  encountered  a  failure  will  not 
necessarily  happen;  a  failure  might  only  occur  during  certain 
machine  states.  The  fault  exposure  ratio  K  is  meant  to  account 
for  both  the  structure  of  the  program  (loops  and  branches)  and 
the  varying  machine  state.  The  value  of  K  can  be  determined  from 
historical  data.  On  the  same  project  or  one  that  is  similar, 
failure  data  is  collected  and  used  to  statistically  estimate  the 
value  of  p .  Then  K  is  obtained  as 


K  = 


JL 

Bf 


(140) 


When  no  information  is  available,  the  industry  average  value  of 
K-4.20X10'7  [Musa  et  al .  (1987)]  is  recommended.  The  section  in 
this  report  on  Prediction  discusses  the  quantities  K,  S,  and  M  in 
detail.  K  is  broken  down  into  two  independent  factors: 


K  *  S  '  M 


(141) 


where  S  accounts  for  structure  and  M  accounts  for  machine  state. 


5. 2. 2. 3  Egtimatlgn 

Growth  testing  takes  place  during  system  test,  once  the  software 
has  been  fully  integrated  and  stress  testing  has  been  completed. 
Upon  the  occurrence  of  each  software  failure,  the  failure 
identification  personnel  (testers)  must  record  the  cumulative 
execution  time  from  the  start  of  growth  testing.  Musa  et  al . 
(1987)  furnish  auxiliary  techniques  for  evolving  software 
(integrated  in  successive  builds)  and  for  situations  in  which 
grouped  failure  data  or  calendar  time  failure  data  is  the  only 
data  available.  Failure  data  is  said  to  be  grouped  when  the  only 
information  available  is  the  number  of  failures  that  occurred  in 
disjoint  time  intervals. 

The  Basic  Execution  Time  Model  provides  general  forms  for  the 
time-dependent  failure  rate  and  the  mean  value  function.  As 
derived  earlier,  the  formulas  are 
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X(x)  •  -&-exp[-px] 
vo 


(142) 


and 


p(t)  »  v0  (1-exp  t-Pr]  ) 


(143) 


The  unknown  parameters  are  p  and  v0.  (It  is  also  possible  to 
parameterize  the  model  in  terms  of  other  quantities.)  Let  the 
observed  failure  times  be  denoted 


Tj.,  T2' 


(144) 


where  n  is  the  number  of  observed  failures.  Let  r  £rn  be  the 
time  at  which  the  test  ends.  In  statistical  quality  control 
terminology,  this  type  of  test  is  called  a  "time-truncated  test." 
All  times  are  measured  in  cumulative  execution  time  from  the 
3tart  of  system  test. 

A  number  of  statistical  estimation  methods  can  be  employed  to 
estimate  the  values  of  p  and  vQ  from  the  failure  times.  one  of 
the  best  general  methods  for  statistical  estimation — and  by  fai¬ 
ths  most  popular — is  the  method  of  maximum  likelihood.  The 
maximum  likelihood  estimators  are  the  values  of  p  and  v0  that 
maximize  the  probability  of  the  observed  outcome  r.,  t2,  . ..,  r  . 
Maximum  likelihood  estimators  exhibit  many  favorable  large-sample 
properties.  They  are  consistent  (variance  tends  to  zero  and 
expectation  tends  to  the  true  population  parameter  as  the  number 
of  observations  increases),  efficient  (lowest  variance),  and 
asymptotically  normal  (so  that  confidence  intervals  can  be  easily 
established) .  Another  property  of  maximum  likelihood  estimators 
is  invariance,  which  means  that  the  maximum  likelihood  estimator 
of  a  function  of  an  estimator  can  often  be  obtained  by 
substitution.  A3  an  example,  if  the  maximum  likelihood  estimator 
of  w  is  0,  the  maximum  likelihood  estimator  of  w3  is  ft3. 

Musa  et  al.  (1987)  derive  the  maximum  likelihood  estimation 
equation  for  p  as 


n  _  nx , 

P  exp[px,]-l 


0 


(145) 
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The  estimate  for  0  is  obtained  by  solving  the  equation  for  0.  The 
maximum  likelihood  estimator  for  v0  is 


vo 


_ n 

1-exp  [-p-rj 


(146) 


In  that  equation  there  is  nothing  to  solve;  the  estimate  for  0 
(already  obtained)  is  simply  inserted  into  the  equation  to  obtain 


The  two  equations  above  provide  point  estimates— -single  values— 
for  the  parameters.  It  is  often  more  informative  to  provide  an 
interval  estimation.  Interval  estimation  provides  a  range  of 
values  within  which  the  true  value  is  asserted  to  lie.  The 
probability  that  a  correct  interval  estimate  is  obtained  is  the 
confidence  coefficient  l-a.  The  interval  is  bounded  by  lower  and 
upper  confidence  limits.  A  confidence  interval  that  has  an 
associated  confidence  interval  of  l-a  is  called  a  100 (l-a) % 
confidence 
limits  for 
information 

information  supplied  by  an  unknown  parameter.  It  is  the 
reciprocal  of  the  variance  of  the  estimator. 

The  Fisher  information  for  0'a  estimator  is 


interval.  The  lower  and  upper  100 ( l-a) %  confidence 
0  are  denoted  0 .  and  £h,Bh  respectively.  The  "Fisher 
"  [Fisher  (1922)]  is  a  measure  of  the  amount  of 


J(0) 


t*exp  [ftt,] 
(exp[pT#]  -l)a 


(147) 


Then  the  lower  and  upper  confidence  limits  are  obtained  as 


and 


K(l-«)/3 


(148) 


Jhigh 


6+ 


(149) 
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where  k(1  )/2  is  the  corresponding  normal  deviate.  The  normal 
deviate  is  found  as  follows:  The  cumulative  distribution 
function  for  the  standard  normal  distribution  is 


Fix) 


X 


(150) 


where  7*3.14159...  and  -«<x<«.  The  standard  normal  deviate 
ko-«)/2  is  that  value  of  x  for  which 

Fix)  -  (l-a/2)  (151) 


Tables  of  F(x)  can  be  found  in  most  statistics  textbooks. 
5-1  provides  some  common  values  for 


Table  5-1.  Standard  Normal  Deviates 


0 

0.2 

1.282 

0.1 

1.645 

0.05 

1.960 

0.02 

2.326 

H 

o 

o 

2.576 

i  0.002 

3.090 

Table 


On  a  programmable  calculator  or  computer,  the  value  of  F(x)  can 
be  computed  to  tie  four  significant  figures  usually  found  in 
tables,  using  the  approximation  formula 


F(x)  ■  l-F/(x)  (a1u+aIu2+a3us) 


(152) 


where 


Wix) 


(153) 


and 
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u 


1 

l+a0x 


(154) 


The  constants  a0,  a,,  a2,  and  a3  are  provided  in  Table  5-2. 


Table  5-2 .  Normal  Approximation  Constants 


Constant 

Value 

an 

0.33267 

0.4361836 

_  a? 

-0.1201676 

_ _ 

0.9372980 

Because  of  the  invariance  property  of  maximum  likelihood 
estimators ,  the  corresponding  confidence  intervals  for  v0  can  be 
established  by  successively  substituting  /Jlow  and  /?hJ  h  into  the 
estimation  equation  for  v0: 


<V0)  Jow 


_ n _ 

l-exp[-PJwT#] 


(155) 


<vo>  high 


_ n _ 

1-expI-P^t,] 


(156) 


5 . 2 . 2 . 4  saaflnsaa  o£  Eli 


It  was  concluded  that  Littlewood's  (1979a)  prequential  likelihood 
statistic  and  u-plot  are  helpful  during  growth  modeling  to 
determine  the  model's  "goodness  of  fit."  When  the  tester  uses 
the  Basic  Execution  Time  Model  or  some  other  time  domain  software 
reliability  model,  it  is  not  enough  to  blindly  apply  the  model. 
The  tester  should  monitor  how  well  the  model  is  fitting  the 
failure  data.  If  the  model  is  not  fitting  well,  then  the  user 
should  switch  to  an  alternative  model,  such  as  the  Logarithmic 
Poisson  Model. 


A  u-plot  is  constructed  according  to  the  following  scheme.  During 
growth  testing,  the  user  employs  a  statistical  inference 
procedure,  such  as  maximum  likelihood,  to  estimate  the  parameters 
of  the  Basic  Execution  Time  Model. 
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The  estimated  cumulative  distribution  function  (Cdf)  is 


fi(x')  -  1  -  exp [-X  (xm)  *']  (157) 


where  rm  is  the  cumulative  execution  time,  X(Tm)  is  the  failure 
rate  at  that  time,  and  r'  is  execution  time  measured  from  the 
present.  When  the  inter  failure  time  r't  is  later  observed,  the 
probability  integral  transform 


<«•) 

is  recorded.  Each  failure  results  in  another  u, .  The 

probability  integral  transform  implies  that  the  u,'s  should  look 
like  a  random  sample  from  a  uniform  distribution  over  the 
interval  (0,1),  if  the  sequence  of  predictions  was  good.  The 
accuracy  of  the  model  with  respect  to  the  particular  program  can 
be  gauged  by  drawing  a  u-plot.  In  a  u-plot  the  sample  cumulative 
distribution  function  of  the  u,'s  is  compared  with  the  cumulative 
distribution  function  of  the  uniform  distribution  over  (0,1), 
visually  or  through  use  of  numerical  goodness-of-fit  measures 
such  as  the  Kolmogorov  distance.  The  specific  points  plotted  to 
form  the  cumulative  distribution  function  are 

<u(1) ,  1/  (r+l) ) 

<utJ).  2/ <*.»>>  (l„, 

•  •  • 


(u(r) , r/ (r+l) ) 


where  u(1),  u<2),  ...,  u(r+1)  are  the  u,'s  rearranged  in  ascending 

order. 


5.2. 2. 5  RggflUbrfltlon 

Brocklehurst  et  al.  (1990)  show  how  the  u-plot  can  be  employed  to 
perform  a  kind  of  "adaptive  modeling."  The  u-plot  showB  how  well 
the  model  is  fitting  the  failure  data.  The  information  in  the  u- 
plot  can  be  used  as  a  feedback  mechanism  to  modify  and  improve 
the  model,  to  "recalibrate"  it.  The  recalibrated  model  corrects 
for  systematic  bias  or  noisiness  that  the  model  is  experiencing 
when  being  used  on  a  particular  program.  The  recalibration  takes 
place  by  applying  a  function  G  (•  )  to  the  estimated  Cdf.  The 
Brocklehurst  paper  only  describes  G*(* )  graphically.  The  study 
team  has  developed  a  recalibration  formula  based  on  the  same 
principles  as  the  graphical  technique. 
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The  function  G*(*  )  is  expressed  as 


f^t)  ♦J(UiM)*U|j))-U„) 
(r+l)  (u(J,1)+u(J)) 


uij)  stsu(i+i) 


U«o> 


where  is  the  j-th  value  when  the  u^'s  are  put  in  ascending 
order  or  magnitude ,  and  r  is  the  number  of  uf's,  with  u(C)*0  and 

To  perform  the  recalibration  the  user  applies  the  transformation 

fl(x)  -  <l«i> 


The  accuracy  of  recalibrated  models  has  been  shown  [Brocklehurst 
£fc_  al.  (1990)]  to  be  generally  better  than  that  of  the  original 
model . 

The  study  team  has  developed  a  computer  program  to  automatically 
create  a  u-plot  and  perform  recalibration  of  the  model. 

5.2.2. 6  Operational  Profile 

During  growth  testing,  the  environment  in  which  the  program 
executes  must  be  controlled.  The  environment  includes  the 
hardware  platform,  the  system  generation  parameters,  and  the 
workload.  An  important  part  of  a  program's  execution  environment 
is  the  operational  profile. 

An  input  state  is  a  set  of  input  variable  values  for  a  particular 
run.  Each  input  variable  has  a  declared  data  type — a  range  and 
ordering  of  permissible  values.  The  set  of  all  possible  input 
states  for  a  program  is  the  input  space.  Each  input  state  is  a 
point  in  the  input  space.  An  operational  profilo  is  a  function 
p('  )  that  associates  a  probability  p(i)  with  eaci  point  i  in  an 
input  space  I.  Since  the  points  in  the  input  sp^ce  are  mutually 
exclusive  and  exhaustive,  all  the  probabilities  must  add  up  to 
one: 


^p(i)  -  l 


(162) 


To  illustrate  these  definitions,  consider  a  program  with  three 
input  variables.  They  are  each  of  data  type  Boolean,  meaning 
that  they  have  two  possible  values:  TRUE  and  FALSE.  The  input 
space  has  eight  points:  (FALSE, FALSE, FALSE) ,  (FALSE, FALSE, TRUE)  , 
(FALSE, TRUE, FALSE) ,  ( FALSE , TRUE , TRUE) ,  (TRUE, FALSE, FALSE) , 
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( TRUE , FALSE , TRUE ) ,  ( TRUE , TRUE , FALSE ) ,  ( TRUE , TRUE , TRUE ) .  Letting 
T  stand  for  TRUE  and  F  for  FALSE,  an  operational  profile  for  the 
program  might  look  like: 


p(FFF)  *  0.1 
p(FFT)  =0.2 
P(FTF)  =0.1 
p(FTT)  =0.3 
p(TFF)  =  0.025 
p(TFT)  =0.02 
p(TTF)  =  0.025 
p(TTT)  =0.05 


(i«3) 


The  distribution  of  input  states  is  thus  established  by  the 
operational  profile. 

The  concept  of  operational  profile  formalizes  what  is  meant  by  a 
consistent  input  environment. 

During  growth  (and  demonstration)  testing  must  be  kept 
stationary:  The  p(i)'s  must  not  change.  The  input  states  chosen 
for  test  cases  must  form  a  random  sample  from  the  input  state  in 
accordance  with  the  distribution  of  input  states  that  the 
operational  profile  specifies. 

The  concept  of  a  stationary  operational  profile  is  a  crucial 
assumption  in  building  software  reliability  models  such  as  the 
constant  failure  rate  model  for  frozen  code  (see  Appendix  B)  and 
the  Coincident  Failures  model  for  multiversion  software  (see 
section  on  Reliability  Allocation) . 

It  is  generally  not  practical  to  fully  express  or  specify  an 
operational  profile,  because  the  number  of  input  states  for  even 
a  simple  program  can  be  enormous.  As  an  example,  if  a  program 
has  three  input  variables,  each  of  which  is  a  3 2 -bit  integer,  the 
number  of  distinct  input  states  is 


2»  .  2ia  •  233  «  2 **  *  7 . 9  x!0ai  (1*4) 


At  best,  the  customer  will  express  or  specify  software  usage  in 
terms  of  end-user-oriented  functions.  For  example,  the  customer 
might  state  that  the  usage  of  the  software  is  40%  user  function 
A,  45%  user  function  B,  and  15%  user  function  C.  To  convert  that 
statement  into  an  operational  profile,  let  u  be  the  input 
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variable  corresponding  to  user  function  selection.  Thus,  u  can 
take  on  the  values  A,  B,  or  C.  The  input  space  can  be 
partitioned  into  three  classes  defined  as  follows: 


CLASS A  =  [ier\iu=A} 

CLASS B  a  (165) 

CLASS e  - 


where  iu  is  the  value  of  the  input  variable  u  in  input  state  i, 
The  probability  of  each  individual  input  state  in  class  A  is 


p(i)  «  .40  x 


\CLASS_A 


(166) 


For  class  B  it  is 


p(i)  -  .45  x 


CLASS  £ 


(167) 


and  for  class  C  it  is 


p(i)  ■  .  15  x 


CLASS  C 


(168) 


where  the  vertical  lines  mean  "number  of  elements." 

Once  the  operational  profile  is  established,  a  procedure  for 
selecting  a  random  sample  of  input  states  is  required,  so  that 
test  cases  can  be  generated  for  growth  testing  and  demonstration 
testing.  The  following  procedure  for  selecting  the  input  states 
is  recommended. 

5 . 2 . 2 . 7  Random -InsutrStats  Sslas-tlcn 

The  strategy  employed  is  to  associate  each  input  state  i  with  a 
subinterval  of  the  real  interval  [0,1]  whose  size  is  equal  to  the 
input  state’s  probability  of  selection  p(i) .  As  an  example, 
suppose  that  there  are  only  three  possible  input  states  and  the 
operational  profile  says  that  state  #1  occurs  28%  of  the  time, 
state  #2  occurs  11%  of  the  time,  and  state  #3  occurs  61%  of  the 
time.  State  #1  should  be  associated  with  the  real  interval 
[0,0.28];  state  #2  should  bo  associated  with  the  real  interval 
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[0.28/0.39];  and  state  #3  should  be  associated  with  the  real 
interval  [0.39/1.0]. 

The  next  step  is,  for  each  test  case  needed,  to  generate  a  random 
number  in  the  interval  r o ,  1  ] .  In  the  example,  if  that  random 
number  falls  in  the  interval  [0,0.28],  input  state  #1  is 
selected;  if  it  falls  in  the  interval  [0.28,0.39],  input  state  #2 
is  selected;  and,  if  it  falls  in  the  interval  [0.39,1],  input 
state  #3  is  selected. 

To  generalize  and  formalize  this  procedure,  suppose  that  the 
input  space  contains  k  input  states.  Further  suppose  that  the 
probabilities  associated  with  the  input  states  are 


px,  p2,  ...,  pk  (169) 


Let 


STARTX  ■  0 


(170) 


and 


J-i 

START j  -  gp1  ,  >2,3,...,*  (171) 

and  let 

END j  «  START M  ,>l,Jc-l  (172) 


and 


ENDk  -  1  (173) 


Each  input  state  is  now  associated  with  a  subinterval 
[START,,  END,]  of  the  interval  [0,1].  The  length  of  the 
subinterval  is  equal  to  the  input  state's  probability  of 
selection. 

Now  a  random  number  uniformly  distributed  in  the  interval  (0,1) 
is  generated.  The  random  number  will  fall  into  exactly  one  of 
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the  intervals  [ START j ,  ENDj] .  The  input  state  so  selected  is 

input  state  j .  j  > 

The  cardinal  rule  for  increasing  the  efficiency  of  growth  and 
demonstration  testing  is  not  to  repeat  the  same  test  case  more 
than  once.  ("Regression  testing"  after  a  program  change  is 
another  matter.)  This  rule  per  se  is  not  that  helpful  because 
the  large  number  of  input  states  that  most  programs  have  means 
that  the  repetition  of  a  randomly  selected  test  case  is  a  rare 
event.  However,  the  nonrepetition  rule  combined  with  the 

technique  of  equivalence  partitioning  can  indeed  increase  testing 
efficiency.  Equivalence  partitioning  was  first  described  by 
Myers  (1979). 

Imagine  a  "test  oracle"  e(i)  that  evaluates  to  1  if  a  run 
starting  from  input  state  i  will  result  in  software  failure 
(crash,  hang,  or  erroneous  output)  and  evaluates  to  0  if  the  run 
will  result  in  success.  Consider  a  set  W  of  input  states  defined 
as  follows: 


W  {ie.r!(Vi  e(i)  -0]  V [Vi  e(i)«l]}'  (174) 


That  is,  W  is  a  subset  of  the  input  space  such  that  all  input 
states  in  the  subset  fail  alike:  If  any  input  3tate  would  cause 
the  program  to  fail,  then  so  would  any  other  input  state  in  the 
subset.  If  the  program  starting  from1  any  input  state  in  the 
subset  would  succeed,  then  the  program  would  also  succeed 
starting  from  any  other  input  state  in  the  subset.  Such  a  subset 
is  called  an  equivalence  class,  of  which  there  may  be  many. 

Testing  personnel  can  determine  equivalence  classes  from  an 
understanding  of  the  progiam  structure  and  logic.  Once  an 
equivalence  class  is  identified,  only  one  representative  input 
state  from  the  class  needs  to  be  tested;  if  a  run  starting  from 
the  representative  input  state  results  in  success,  then  it  can  be 
concluded  that  runs  starting  from  all  members  of  the  class  would 
result  in  success.  If  a  run  starting  from  the  representative 
input  state  results  in  failure,  then  it  can  be  concluded  that 
runs  starting  from  all  members  of  the  class  would  fail. 

The  input  states  that  are  members  of  an  equivalence  cxass  are 
removed  from  the  operational  profile  and  replaced  by  their  one 
representative  input  state.  The  probability  associated  with  the 
representative  input  state  is  assigned  the  sum  of  the 
probabilities  of  the  members  of  the  equivalence  class. 

Since  the  probability  of  selection  of  the  representative  of  an 
equivalence  class  is  a  sum,  it  can  be  relatively  large  compared 
to  individual  input  states,  and  consequently  the  representative 
will  likely  be  selected  more  than  once  during  testing.  The 
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second  and  subsequent  times  the  representative  is  selected,  the 
test  dees  not  have  to  be  re-run,  only  the  results  from  the 
original  run  recounted. 

5 . 2 . 2 . 8  Software  Reliability  Growth  Management 

The  demonstrated  reliability  is  an  assessment  of  the  currant 
reliability  of  the  software.  It  is  obtained  by  recording  the 
cumulative  execution  times  at  which  failures  have  occurred  so  far 
during  system  test.  The  operational  profile  from  which  the 
random  test  cases  were  selected  has  to  have  been  stationary 
during  the  testing  and  representative  of  the  program's  field  use. 
The  statistical  estimation  technique  presented  previously  is 
employed  to  obtain  estimates  of  the  Basic  Execution  Time  Model's 
0  and  v0  parameters.  Then  a  number  of  derived  quantities  [Musa 
(1987)]  can  be  obtained  to  support  reliability  growth  management. 
The  demonstrated  reliability  is  then  given  by 

R(t)  -  exp  [-1  (t#)  t]  (175) 


where  r0  is  the  point  in  cumulative  execution  time  at  which  the 
assessment  is  made.  The  projected  reliability  for  Ar,  execution 
time  units  from  now  is 


£<t)  -  exp.[-A(T#+At#)t]  (175) 


The  additional  execution  time  required  to  reach  a  failure  rate 
objective  A,  can  be  computed  from 


At. 


P 


(177) 


The  additional  number  of  expected  failures  to  reach  the  objective 
is  given  by 


An  -  P  (A  (t#)  -Ajr] 


(178) 


During  system  test,  project  management  should  expect  that  certain 
levels  of  reliability  will  be  attained  at  certain  milestones,  to 
assure  that  reliability  growth  is  progressing  at  a  sufficient 
rate  to  meet  the  ultimate  reliability  requirement. 
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Since  the  tine-dependent  failure  rate  is 


1  (t)  *  exp[-(Jx] 
vo 


(179) 


the  reliability  growth  curve  can  be  plotted  as  a  straight  line  on 
semi-log  paper  as 


lnl  ■  ln-£-  -  Pt  (180) 

vo 


The  next  section  discusses  how  to  relate  execution  tine  to 
calendar  tine. 


5. 2. 2. 9  Calendar  Tine  Modeling 

Early  software  reliability  models  used  calendar  tine  (or  did  not 
specify  what  type  of  time  was  being  measured) .  Musa  introduced 
the  idea  of  execution  time  as  the  failure-inducing  stress  on  a 
program.  Software  can  only  fail  when  it  is  executing.  This 
simple  primal  idea  resulted  in  increased  accuracy  in  software 
reliability  modeling  [Trachtenberg  (1985),  Musa  and  Okumoto 
(1984),  Hecht  (1981)].  concentrating  on  execution  time,  though, 
left  a  void.  Project  management  and  even  software  engineers 
usually  think  in  terms  of  calendar  time.  Project  deadlines  and 
milestones  are  all  expressed  in  terms  of  calendar  time.  Project 
personnel  are  paid  for  calendar  time.  To  fill  this  void,  Musa 
added  a  "calendar  time  component"  to  his  Basic  Execution  Time 
Model.  The  calendar  time  component  provides  the  ratio  between 
calendar  time  and  execution  time.  The  model  takes  into  account 
the  constraints  involved  in  applying  personnel  and  computer 
resources  to  the  software  development  project  during  system  test. 

The  available  quantities  of  failure  identification  (testing) 
personnel,  failure  resolution  (debugging)  personnel,  and  computer 
time  are  considered  to  be  constant  during  system  test  because  of 
the  long  lead  times  usually  required  for  training  and 
procurement . 

At  any  point  in  time  during  system  test,  one  of  the  three 
resources  will  be  limiting  and  will  determine  the  execution 
time/ calendar  time  ratio.  Typically,  system  ter.t  consists  of 
three  consecutive  resource-limited  segments:  (1)  a  failure 
resolution  personnel  limited  segment,  (2)  a  failure 
identification  personnel  segment,  and  (3)  a  computer  limited 
segment  [Musa  et  al.  (1987)]. 
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Segment  #1  is  limited  by  the  number  of  failure  resolution 
personnel,  because  they  cannot  keep  up  with  the  large  number  of 
failures  that  occur  in  a  short  period  of  time  when  X(t)  is  high. 
Jn  segment  #2,  the  failure  identification  personnel  become  the 
bottleneck  as  they  become  fully  occupied  in  testing  and  analysis 
of  the  results.  In  segment  #3,  the  intarfailure  times  grow 
longer  and  longer  and  the  bottleneck  is  the  availability  of 
computer  time. 

Various  quantities  in  the  model  have  an  index  subscript  r 
appended  to  identify  the  resource  referenced: 

r>F:  failure  identification  personnel  (people  who  find  the 

causative  faults  behind  the  software  failures) 

r*I:  failure  identification  personnel  (people  who  run  test 

cases  and  watch  for  failures) 

r**C:  computer  time 

The  usage  %r  of  resource  r  is  a  function  of  cumulative  execution 
time  r  and  the  expected  number  of  failures  experienced  n(r) : 

Xr  *  6^  ♦ 


Each  resource  r  has  an  execution  time  coefficient  6r: 

0F:  average  failure  identification  work  expended 

execution  time  (-0) 

6,:  average  failure  identification  work  expended 

execution  time 

6C:  average  chargeable  computer  time  expended  per  hour  of 

test  execution  time 


(181) 

per  unit 
per  unit 


Each  resource  k  has  a  failure  coefficient  ^r: 

4f:  average  failure  resolution  work  required  per  failure 
/*,:  average  failure  identification  work  required  per 

failure 

Hc:  average  chargeable  computer  time  required  per  failure 

Work  is  typically  measured  in  person-hours  and  computer  time  in 
hours.  Hours  would  ordinarily  be  divided  into  eight  hours 
segments  to  allow  conversion  to  standard  shifts  and  workdays. 

The  change  in  resource  usage  per  unit  of  execution  time  is  given 
by  differentiating  xr  with  respect  to  execution  time: 
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8r  +  M,(t) 


(182) 


The  calendar  time/ execution  tine  ratio  in  each  resource-limited 
segment  is  given  by  the  maximum  rate  among  the  three  resources  in 
that  segment: 


where  Pr  is  the  amount  of  resource  r  available: 

PF:  number  of  available  failure  resolution  personnel 
Pt:  number  of  available  failure  identification  personnel 
Pc:  available  computer  time  (number  of  prescribed  work 
periods,  e.g.,  40 -hour  weeks) 

and  pr  is  the  resource  utilization  factor: 

pF:  failure  resolution  personnel  utilization  factor 

p,  -  1 

.  pc:  computer  utilization  factor 

The  boundaries  of  the  resource-limited  segments,  in  terms  of 
failure  rate  values,  are  given  by  the  following  formula.  It 
provides  potential  transition  points,  which  must  be  individually 
checked  for  plausibility. 


A  .  P*P*^r~PzP^c 
”  "  Prprn,-P,p,| iz 


z*s 


(184) 


If  rt  and  r2  are  two  points  of  cumulative  execution  time  with  a 
resource-limited  segment  limited  by  resource  r,  the  increment  of 
calendar  time  between  the  two  points  is  given  by 


Atr 


*rPrP 


erln 


^(T,) 

1  (t2) 


(185) 


If  the  interval  [r^tj]  spans  more  than  one  segment,  the 
increment  of  calendar  time  should  be  calculated  separately  for 
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each  subinterval  that  lies  in  a  different  segment  and  the  results 
added  together. 

For  growth  management,  project  management  can  influence  the 
resource  quantities  Pc,  PM  and  P,  to  stimulate  software  growth. 
Because  of  long  lead  times  for  training  and  procurement,  the 
resource  quantities  are  usually  constant  during  system  test, 
except  perhaps  for  personnel  overtime.  Overtime  is  only 
effective  for  a  limiting  resource. 
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6 .  RELIABILITY  DEMONSTRATION 


In  fulfillment  of  SOW  Task  5,  techniques  for  formally 
demonstrating  the  achievement  of  specified  reliability 
requirements  for  software  products  and  combined  hardware/ 
software  systems  were  developed. 

6.1  Approach 

A  reliability  demonstration  test  is  an  experiment  conducted  to 
determine  whether  an  item  has  achieved  a  specified  level  of 
reliability.  One  problem  with  such  a  test  is  that  the  results 
are  subject  to  chance  variation.  Occasionally  the  test  results 
will  turn  out  especially  "lucky,"  allowing  a  bad  item  to  be 
accepted,  or  especially  "unlucky,"  allowing  a  good  item  to  be 
rejected.  By  designing  tha  test  using  sound  statistical 
principles,  the  risks  of  accepting  a  bad  item  or  rejecting  a  good 
item  can  be  quantified  and  contained. 

To  design  reliability  demonstration  tests,  it  is  necessary  to 
determine  an  appropriate  time  domain  reliability  model  for 
software  products  and  combined  hardware/software  systems.  In 
particular,  the  applicability  of  the  exponential  (constant 
failure  rate)  model  was  investigated  (SOW  para.  4. 1.5. 2). 

Three  types  of  MIL-STD-781-like  reliability  demonstration  tests 
were  developed:  fixed-length  test,  minimum  failure-free  execution 
period  test,  and  sequential  test.  Since  software  reliability 
demonstration  tests  sometimes  have  to  be  performed  concurrently 
with  growth  testing,  how  to  adapt  those  same  tests  for  periods 
during  which  the  software  is  subject  to  repair  activity  is 
discussed. 

Statistical  random  testing  procedures  for  use  during  software 
reliability  demonstration  testing  are  the  same  as  those  described 
in  the  section  of  this  report  on  Reliability  Growth  Testing 
(Section  4).  The  test  cases  must  be  selected  randomly  in 
accordance  with  a  stationary  operational  profile  that  represents 
anticipated  field  usage.  (SOW  para.  4. 1.5.3)  An  operational 
profile  is  the  probability  distribution  of  the  various  possible 
input  states.  When  a  combined  hardware/software  system  is 
tested,  the  environmental  specification  is  generally  derived  from 
the  specified  mission  profils.  The  operating  conditions  will 
normally  include  thermal  stress,  electrical  stress,  vibration, 
and  humidity.  MIL-HDBK-781  addresses  test  sample  size  and  burn- 
in  requirements  for  hardware. 
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6.2  Results 

6.2.1  Test  Brassfliigsa 


The  test  environment  must  be  stipulated  or  agreed  on.  The 
environment  for  software  includes  the  hardware  platform,  the 
operating  system,  system  generation  parameter  settings,  and  the 
operational  profile.  The  operational  profile  associates  each 
point  in  the  input  space  with  a  probability  of  occurrence.  The 
system  workload  must  also  be  specified.  The  important  thing 
about  the  environment  is  that  it  must  be  representative  of  the 
conditions  the  software  will  experience  in  field  use,  if  the  test 
is  to  have  any  validity. 

Let  the  true  failure  rate  of  the  software  be  denoted  1.  A 
demonstration  test  plan  has  two  failure  rate  parameters: 

The  "upper  test  failure  rate"  10  is  typically  the  failure  rate 
the  customer  "wants"  or  requires.  If  the  producer  produces  a 
product  that  just  barely  meets  the  failure  rate  X0,  then  the 
producer  is  taking  a  big  risk  that  chance  variation  will  result 
in  the  demonstration  test  failing.  To  reduce  that  risk  the 
producer  must  produce  a  product  with  a  lower  failure  rate, 
called  the  "lower  test  failure  rate."  It  should  be  mentioned 
that  MIL-STD-781  uses  mean  time  between  failures  (MTBF)  —the 
reciprocal  of  failure  rate — and  so  the  upper  and  lower  points  are 
reversed.  MTBF  is  a  problematical  concept  in  software 
reliability  because  if  there  is  a  nonzero  probability,  however 
slight,  of  a  particular  program  having  zero  faults,  the  program's 
MTBF  will  be  infinite  [Littlewood  (1975)].  Consequently,  in 
this  section,  only  failure  rates  are  used. 

The  producer's  risk  a  is  the  probability  of  rejecting  a  software 
product  or  hardware/software  system  whose  true  failure  rate  is 
equal  to  10.  The  consumer's  risk  /3  is  the  probability  of 
accepting  software  whose  true  failure  rate  is  equal  to  Xr  (SOW 
para.  4.1.5. 3) . 

A  statistical  test  that  answers  a  yes/no  question  is  called  a 
"test  of  hypothesis."  Specifically,  a  reliability  demonstration 
test  tests  the  simple  null  hypothesis 


H0i  1  -  10  (186) 

(accept  the  item)  against  the  alternative  hypothesis 

Jfx:  A.  -  Xi  (187) 


(reject  the  item) .  In  terms  of  these  hypotheses, 
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a  «  Prfffi  accepted  \hq  true} 


(188) 


and 


P  ■  Pr  [H0  accepted^  true} 


(189) 


To  develop  demonstration  taste,  it  is  necessary  to  choose 
appropriate  time  domain  reliability  models  for  software  products 
and  combined  hardware/software  systems. 

During  a  software  reliability  demonstration  test,  the  code  is 
frozen,  that  is,  it  is  not  modified  to  remove  faults  or  for  any 
other  reason.  The  reason  is  that  any  modification  can  introduce 
new  faults.  Frozen  code,  subj acted  to  a  stationary  operational 
profile,  can  reasonably  be  modeled  as  having  a  constant  failure 
rate.  (See  Appendix  B) .  Multiple  discrepancies  appearing  at  the 
same  time  need  to  be  counted  as  a  single  failure,  because  the 
random  variable  of  interest  is  time  to  first  failure. 

For  a  combined  hardware/ software  system,  the  model  is  also  a 
constant  failure  rate,  based  on  the  following  reasoning.  MIL- 
STD-781  applies  to  demonstrating  the  reliability  of  simple  series 
systems  as  well  as  "complex  maintained  electronic  equipment" 
[O'Connor  (1985)],  so  the  hardware  during  its  "useful  life" 
period  is  already  modeled  by  a  constant  failure  rate.  According 
to  Appendix  B,  software  whose  code  is  frozen,  being  subjected  to 
a  stationary  operational  profile,  can  reasonably  be  modeled  as 
having  a  constant  failure  rate.  According  to  the  section  in  this 
report  on  Reliability  Combination,  if  the  software  executes 
concurrently  and  in  series  with  the  hardware,  then  the  combined 
hardware/software  system  can  be  modeled  by  a  constant  failure 
rate  that  is  the  stun  of  the  hardware  failure  rate  and  the 
software  failure  rate.  Therefore,  the  demonstration  test  plans 
developed  in  this  section  apply  to  both  software  products  and 
most  combined  hardware/software  systems  (SOW  para.  4. 1.5. 2). 

Under  the  constant  failure  rate  model,  the  number  of  failures 
during  a  time  interval  [0,r]  obeys  Poisson's  law,  which  states 
that  the  probability  of  exactly  i  failures  occurring  in  a  time 
interval  of  length  r  is 


J»,  -  exp[-At]-ii^i  (190) 
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The  probability  of  n  or  favor  failures  occurring  during  that  sans 
period  is  given  by  the  Poisson  cumulative  distribution  function 
(Cdf) 


ftei,.)  .  V  (U)^xp[-UJ  (1»1) 

1*0  ^ 


6.2.2  Fixed-Lenoth  Test 

A  fixed  length  software  reliability  demonstration  test  plan 
[Singpurvalla  (1985a)]  provides  a  predetermined  test  duration  and 
an  acceptable  number  of  failures  the  software  is  allowed  to 
accumulate  during  that  test  time.  The  test  is  terminated  with  a 
decision  to  accept  or  reject.  The  statistical  basis  of  the 
fixed-length  test  derives  from  Epstein  and  Sobel  (1953). 
Suppose  that  the  acceptance  criterion  is  x  or  fewer  failures  in 
time  r.  If  the  true  failure  rate  is  A0  then  F(a;l0,r)  is  the 
fraction  of  time  that  x  or  fewer  failures  will  occur.  However, 
in  [l-F(x;A0,  r)  ]  fraction  of  the  time,  greater  than  x  failures 
will  occur.  When  greater  than  x  failures  occur  during  the  test, 
the  test  will  reject  the  software  even  though  the  true  failure 
rate  is  X0.  To  limit  the  probability  of  rejecting  good  software, 
therefore,  x  and  r  must  be  chosen  such  that 


1-.F<x;A.0,t)  &  a  (192) 


Likewise,  if  the  true  failure  rate  is  A,,  then  F(x,A,,r)  is  the 
fraction  of  time  that  the  software  will  be  accepted,  even  though 
the  true  failure  rate  is  A,.  To  limit  the  probability  of 
accepting  bad  software,  x  and  r  must  be  chosen  such  that 


F(xjXlt x)  i  p  (193) 


Thus,  x  and  r  must  simultaneously  satisfy  both  Equations  7  and 
Equation  8. 

A  test  plan  provides  x  and  r  values  as  a  function  of  failure 
rates  10  and  A,,  and  risk  levels  a  and  £.  Since  time  units  are 
arbitrary,  a  test  plan  can  be  made  more  general  by  replacing  10 
and  11  by  their  ratio  d-A,/A0,  called  the  discrimination  ratio. 
Then,  for  example,  the  same  test  plan  will  be  valid  for  the 
combination  A»-o. 000006  and  1,-0.000002,  and  the  combination 
A0-O.OO45  and  1,-0.0015.  For  this  scheme  to  work,  the  time  unit 

is  chosen  so  that  A0«l.  Hence,  A,-d.  As  a  typical  example,  if 

the  original  time  unit  was  seconds,  then  the  new  time  unit  will 
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bo  Ag'1  seconds.  The  test  time  r  is  multiplied  by  Ag*1  to  convert 
to  seconds. 

A  test  plan  provides  an  accept  number  x.  If  x  or  fewer  failures 
are  observed  during  the  test  period,  the  software  is  accepted. 
The  reject  number  is  r»x+l.  If  r  or  more  failures  are  observed, 
the  software  is  rejected.  To  design  the  test*,  the  smallest  x  and 
r  values  need  to  be  found  such  that  simultaneously 


exp  [-t]  -jy  a  1  -  a  (194) 

(because  A0«l)  and 

exp[-td]^il^i  £  p  (193) 


(because  A^d) . 

For  determining  the  reject  number  r,  one  can  exploit  the 
relationship  between  the  Poisson  and  chi-square  distributions. 

The  cumulative  distribution  function  of  the  chi-square 
distribution  is 


i)] 


P(X2|v) 


2»/ar 


A 

/ 


(t) 


i-1 


exp|-- 


cft  ,  (osx*<°°) 


(196) 


where  for  an  integer  argument  n,  r(n+l)«nl.  The  parameter  v  is 
the  number  of  degrees  of  freedom  (i.e.,  the  number  of  free 
variables  entering  into  the  statistic) .  The  values  of  the  chi- 
square  Cdf  are  tabulated  in  many  statistics  textbooks. 
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on  a  programmable  calculator  or  computer  a  series  expansion 
[Abramowitz  and  Stegun  (1970)]  can  be  utilized: 


■P(Xa!v) 


1  + 


(197) 


The  relationship . between  the  Poisson  and  chi-square  distributions 

is 


l-P(Xa|v) 


Sexp‘-"177 


c»~-  ,m«xa/2,  veven 
2 


(198) 


So  to  determine  r  one  finds  the  smallest  integer  r  such  that 


d  * 


%ar,p 

Xar,i-« 


(199) 


where  x2y,x  is  that  xa  value  which  solves  the  equation 

z  ■  p<xa|y)  (aoo) 

Then  x«r-l  is  inserted  into  the  equation 

exp [«t]  £ ■  1-a  (201) 


Equation  16  is  then  solved  for  r.  Call  this  solution  ra.  Then  x 
is  inserted  into 


exp  [-t  dig  (202) 
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which  is  also  solved  for  r.  Call  this  solution  rb.  Because  of 
the  discrete  nature  of  the  distribution,  rt  and  rb  can  be 
expected  to  differ  slightly.  Talcing  r  to  be  the  average 
r-(r11+rb)/2  is  a  reasonable  compromise.  The  a  and  0  risk  levels 
will  change  slightly.  Inserting  x  and  r  back  into  Equation  16 
and  17  and  solving  for  a  and  0  will  provide  the  actual  risk 
levels.  The  original  a  and  0  used  to  construct  the  test  are 
called  the  nominal  risk  levels. 

The  probability  of  the  fixed-length  test  accepting  the  software 
is  given  by 


P0.xa)  -  expt-lf]^  (203> 


This  yields  a  family  of  operating  characteristic  (OC)  curves. 
The  OC  is  a  plot  of  the  probability  of  acceptance  versus  the 
failure  rate:  The  steeper  the  slope  of  the  OC  curve,  the  greater 
the  efficiency  of  the  test  in  discriminating  between  items  of 
differing  reliability. 


A  special  case  of  the  fixed-length  test  is  the  one  in  which  the 
acceptable  number  of  failures  x-0.  The  probability  of  zero 
failures  in  a  test  of  duration  r  is 


P0. o  -  exp  [-It]  Jp  1  -  exp  [-It]  (204) 


Solving 


exp[-l0t]  i  l  -  a  (205) 


and 


exp  [-lit]  i  P  (206) 

gives 
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lGt  -  -ln(l-a);  X:t  ■  -ln|J 


(207) 


In  terns  of  the  decision  risks,  the  discrimination  ratio  is 

_  InB  m  ^ 

T;  V  lull-.)  4«* 


(208) 


This  discrimination  ratio  can  be  quite  large.  For  instance,  in 
the  case  a-0.15,  £-0.15,  the  producer  is  saddled  with  a 
discrimination  ratio  of  11.67.  This  discrimination  ratio  is 
high;  it  is  usual  to  design  to  a  discrimination  ratio  on  the 
order  of  1.5  to  3.  A  less  stringent  type  of  failure-free  test 
will  now  be  described,  in  which  the  required  discrimination 
ratios  are  lower. 

In  a  minimum  failure-free  period  life  test  [Angus  et  al.  (1985)], 
the  item  is  given  a  time  limit  of  T  time  units  to  achieve  a 

failure  free  interval  of  t  time  units.  The  null  hypothesis  is 

IL:  1-10,  and  the  alternative  hypothesis  is  H,:  1-1. .  Note  that 

tne  null  and  alternative  hypotheses  are  reversed  from  those  of 

the  fixed-length  test.  A  test  plan  can  be  designed  for  any 
discrimination  ratio  d<dmM.  Let  the  function  Fut  be  the 
cumulative  distribution  function  of  the  waiting  time  to 
completion  of  the  failure-free  period.  Renewal  theory  [Feller 
(1966)]  provides 


( w;  1  it) 


exp [-11  £ 

0, 


(lv) 


n-1 


(n-l)  I 


flr(n)  ,  WiOO 


w<  t 


(209) 


where 

?<n>  <2i°’ 


and 
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u?  ■  1,  u  >  0;  u*  *  0,  u  i  0;  u.*  *  [max(u,  0)  ] A  (211) 


The  function  g(n)  represents  the  probability  that  (n-1)  points, 
chosen  randomly  in  the  interval  (0,w) ,  partition  the  interval 
into  parts  all  of  which  are  of  length  less  than  or  equal  to  t. 

Test  plans  can  be  constructed  by  fixing  the  ratio  t/T  and 
iteratively  finding  A0t  and  A,t  so  that 

l  -  a  -  FVg(TiX 0,  e) ;  (£12) 


and 


P  -  F„t(T:Xlt  t)  (213) 


6.2.4  atfluintlal  Teat 

In  a  fixed-length  test,  a  failed  test  can  be  stopped  just  as  soon 
as  the  reject  number  r  is  reached.  In  a  failure-free  execution 
period  test,  a  successful  test  can  be  stopped  once  the  failure- 
free  interval  is  achieved. 

A  sequential  test  can  be  stopped  early  when  the  test  reveals  its 
true  character  [Wald  (1947),  Epstein  (i954) ,  Singpurwalla 
(1985b)].  The  item  is  operated  and  the  cumulative  number  of 
failures  and  time  on  test  are  continuously  monitored. 

Let  the  null  hypothesis  be 


HQ:X  *  X 0  (214) 


and  let  the  alternative  hypothesis  be 


HiiX  -  Xx  (215) 


Let 


A  ■  (216) 

a 
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and 


B  • 


JL 

1-a 


(217) 


Let  p0  be  the  probability  of  a  software  failures  in  time  r  if 
A«A0,  and  p1  be  the  probability  of  a  failures  in  time  r  if  A-A^ 
Then  the  sequential  probability  ratio  is 

Pi  u  •xp[-A1t]  (k1x)x  j  expI-ApTj  (Apt)*  ^ 

Po  x i  '  /  xl 


•  d*ex p f  (A0  -  Ax>  t] 


(219) 


where  x  is  the  total  number  of  failures  observed  by  time  r. 
Testing  continues  as  long  as  the  inequality 


B  <  h.  <  a 
Po 


(220) 


holds.  Then,  when  p,/p0  S  B  for  the  first  time,  testing  stops 
and  Hg  is  accepted.  When  p./pfi  2:  A  occurs  for  the  first  time, 
testing  stops  and  H0  is  rejected. 

Graphically,  the  "continue  test"  region  is  bounded  by  two 
straight  lines  with  common  slope  s  and  intercepts  -h1  and  h0. 

The  x-axis  is  the  cumulative  number  of  failures  k,  and  the  y~axis 
is  the  time  on  test  r. 

Solving  for  r  gives 


hx  +  ks  <  t  <  ,h0  +ks 


(221) 


where 


5 


lnd 

t-t; 


(222) 
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(223) 


*o 


-Ins 

\l“*0 


and 


(224) 


The  acceptance  time  line  is  r  -  +  ks.  The  moment  the  plot  of 
failures  k  versus  time  r  crosses  thic  line,  the  software  can  be 
accepted.  The  rejection  tine  line  is  -h,  +  ks.  The  moment  the 
plot  of  failures  k  versus  time  r  crosses  that  line,  the  software 
can  be  rejected.  As  long  as  the  staircase  plot  of  failures  k 
versus  time  r  stays  between  the  two  lines,  the  test  continues. 

In  terms  of  x,  the  rule  is 


t  (Xq  -  X^)  ~lnA  x  ( Xq—Xj )  “In B 

ITnd  ^Lnd 


(225) 


Because  a  sequential  test  can  theoretically  go  on  indefinitely, 
it  is  desirable  to  stop  ("truncaten)  the  test  at  some  point  to 
provide  a  reasonable  maximum  test  time.  Truncation  will  tend  to 
increase  the  decision  risks  a  and  0,  but  research  [Epstein 
(1954)]  has  shown  that  the  risks  are  not  increased  significantly 
if  the  test  is  truncated  at 


r0  ■*  3r 


(225) 


where  r0  denotes  truncation  number  of  failures  and  r  is  the 
reject  number  for  the  corresponding  fixed-length  test.  Recall 
that  r  is  the  smallest  integer  such  that 


d  *  ~-r'-  (227) 

Xar.l-. 
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Th«  truncation  operating  time  r0  is  then 


*o  -  s  •  r0 


(228) 


where,  as1  before,  s  is  the  common  slope  of  the  decision  lines 


5  ■ 


lnd 


(229) 


The  demonstration  test  can  be  terminated  and  the  software 
accepted  if  k<r0  and 


x  *  min(h0+ics,x(j) 


(230) 


The  test  can  be  terminated  just  as  soon  as 

X  i  +  ks  (231) 


or  k»r0  and 


x  <  *0 


(232) 


6.2.5  Pamongtr at  ion  JTtf  t  Ang-dm  ing.J3rs.vth..  icagt  inq 

Consider  the  counting  process  (N(r),r  2  0},  where  N(r)  is  the 
number  of  software  failures  that  occur  by  cumulative  execution 
time  r.  In  Musa  et  al.'s  (1987)  software  reliability  growth 
models  (the  Basic  Execution  Time  Model  and  the  Logarithmic 
Poisson  Model) ,  software  failures  are  generated  according  to  a 
nonhomogeneous  Poisson  process  (NHPP) .  The  probability  of  k 
failures  occurring  by  time  r  is 


Pk(x)  ■  Pr (i\T(x)  ■£}  »exp[-n(x)]  too  (233) 
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where 


H(t)  *E{W(t)}  »  jk(x)dx  (234) 


ie  the  "mean  value  function"  and  A (r)  is  the  tine-dependent 
failure  rate.  In  particular,  the  Basic  Execution  Tine  Model  is 
characterized  by  the  nean  value  function 


H(t)  -  v0(l-exp [-pt] )  (235) 


and  failure  rate 

A(t)  -  m  pv0exp[-pt]  (236) 


where  0  and  v0  are  the  parameters  of  the  nodel. 

The  techniques  for  reliability  demonstration  described  in  this 
section  are  based  on  the  assumption  of  a  constant  failure  rate, 
which  implies  a  homogeneous  Poisson  process  (HPP) .  When  software 
is  subject  to  repair,  one  expects  that  the  softwara  will  exhibit 
a  decreasing  failure  rate.  Barlow  and  Proschan  (1967)  showed 
that,  in  a  constant  failure  rate  fixed-length  test,  a  decreasing 
failure  rate  favors  the  consumer.  Fortunately,  a  transformation 
of  the  time  scale  reduces  an  NHPP  to  an  HPP,  as  shown  below. 
Then,  the  methods  for  fixed-length,  failure- free  execution 
period,  and  sequential  test  design  can  be  used  without  change. 

Let  the  new  time  variable  be  denoted  u,  and  define  it  by 


u  ■  Mt) 


(237) 


That  is,  transform  time  r  into  the  mean  value  function  evaluated 
at  time  r. 

Consider  the  stochastic  process 


{M(u) ,  mo} 


(238) 
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where 

M{u)  «  tf(n_1(u) ) ,  UiO  <239) 

The  mean  value  function  Is 

E{M(u)}  -  R{N{\i‘l(u) ) }  «  p(»i“Mu) )  *U  <240) 

and  the  failure  rate  is 

X(U)  =  -  l  <24l) 

du 

Since  the  rate  parameter  is  a  constant,  M  is  an  HPP.  Thus 

P*(u)  ■  Pr  {M(u)  — Jc)  -exp[-u]~£  <242) 


To  transform  a  time  on  the  u  scale  back  to  the  original  r  scale, 
the  inverse  relationship 


t  -  p*l(u) 


<243) 


is  employed.  For  the  Basic  Execution  Time  Model, 

ln(l--S.] 

n'Mu)  -  — P»v0>o 


<244) 


The  exponent ial/HPP  model  of  frozen  coda  is  completely  specified 
by  the  failure  rate  parameter  1.  The  NKPP  model  of  code  under 
repair  requires  two  parameters,  0  and  v„.  (It  is  also  possible 
to  parameterize  the  model  in  terms  of  other  quantities  such  as 
X0,  w0,  but  two  parameters  will  always  be  needed.) 

In  summary,  the  values  of  0  and  v  are  obtained  through  the 
parameter  estimation  techniques  described  in  the  section  of  this 
report  on  Reliability  Growth  Testing.  Next  an  appropriate  fixed- 
length,  failure-free  execution  interval,  or  sequential  test  is 
selected.  Then  each  failure  time  experienced  r,  is  transformed 
into  u,«m(t,)  in  determining  the  results  of  the  test. 
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6.2.6 


Demonstration  testing  of  highly  reliable  software  can  be  time- 
consuming  (the  fixed-length  test  time  for  given  set  of  a,  0,  and 
d  parameters  is  proportional  to  Ig’1)  .  Another  time-consuming 
situation  is  when  10  and  A.1  are  close  together.  One  solution  is 
to  test  multiple  copies  of  the  software  simultaneously.  Each 
copy  must  be  identical  and  must  be  subjected  to  test  cases 
randomly  selected  from  the  same  operational  profile.  The  test 
cases  for  each  copy  must  be  selected  independently.  The  total 
time  on  test  V(t)  is  the  sum  of  the  individual  times  on  test  of 
each  copy.  When  a  copy  fails,  it  is  restarted  (that  is,  this  is 
a  test  with  replacement) .  Wherever  time  on  test  r  appears  in  the 
formulas  for  fixed-length,  minimum  failure-f ree  execution  period, 
and  sequential  tests,  it  can  be  replaced  with  V(t),  where 


V(t)  ■  g*i(C)  (245) 


is  the  total  time  on  test  up  to  (wall  clock)  time  t,  N  is  the 
number  of  copies,  and  x,  (t)  is  the  time  on  test  of  copy  i  up  to 
(wall  clock)  time  t. 
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7.  VALIDATION 

In  fulfillment  of  SOW  Task  6,  the  models  and  techniques  for 
software  reliability  prediction,  growth,  and  demonstration  were 
validated  on  actual  projects. 

7.1  Appcaash 

The  techniques  are  based  on  several  models  for  software  not 
subject  to  repair,  a  model  for  software  subject  to  repair,  and  a 
set  of  models  for  software  reliability  prediction.  When  software 
is  not  subject  to  repair,  the  study  employs  a  constant  failure 
rate  model  (discussed  in  Appendix  B) .  When  software  is  subject 
to  repair,  the  study  employs  a  nonhomogeneous  Poisson  process, 
the  Basic  Execution  Time  Model.  For  software  reliability 
prediction,  the  study  uses  models  that  were  developed  based  on 
regression  analysis  of  measurable  product/process  characteristics 
(metrics) . 

The  techniques  based  on  these  models  were  used  on  ongoing 
projects  to  assess  workability  and  accuracy.  The  feedback  from 
the  projects'  personnel  was  exploited  to  refine  the  techniques 
and  the  presentation  of  the  techniques  in  the  draft  military 
handbook.  Since  the  models  and  techniques  were  validated  in  one 
organization,  it  is  recommended  that  the  validation  be  repeated 
in  a  wide  variety  of  organizations. 


7.2 

The  first  model,  which  will  be  designated  Model  A,  is  developed 
axiomatically  in  Appendix  B  and  assumes  a  constant  failure  rate 
for  frozen  code  subjected  to  a  stationary  operationally  profile. 
This  model  forms  that  basis  for  software  and  hardware/ software 
relibility  demonstration  tests. 

The  second  model,  which  will  be  designated  Model  B,  is  the  Basic 
Execution  Time  Model  for  software  going  through  system  test. 

7.2.1  Mfl&ti  ,  ft -.If  am  ..Cofli  1 


The  study  team  first  tested  the  validity  of  the  assumption  that 
the  underlying  time-to-failure  distribution  is  exponential.  The 
assumption  of  an  exponential  distribution  of  failure  times  is 
equivalent  to  an  assumption  of  constant  failure  rate  or  the 
assumption  that  failures  are  generated  by  a  Poisson  process.  The 
cumulative  distribution  function  (Cdf)  of  the  exponential 
distribution  is  given  by 


Fix)  ■  0,  T  <  0 

»  l-exp [-Xt] ,  t  z  0 


(246) 


Suppose  that  the  interfailure  times  aro 
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(247) 
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t 
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A  graphical  procedure  [Epstein  (1960) ]  for  testing  the  validity 
of  the  exponential  asumption  is  to  put  the  interfailure  tines 
into  ascending  order 

t(D  *  x(3 )  s  £  T(n)  (248) 


Forn  the  empirical  cdf 


*<*««> 


i 

n+1 


(248) 


and  plot  the  points 

(t(i)  /In  [1/  (l-F(t{i))  ] ) ,  i-l,2 (250) 

If  the  exponential  assumption  holds,  then  the  plotted  points  will 
form  a  straight  line  passing  through  the  origin.  The  slope  of 
the  line  will  be  the  failure  rate  X.  The  graphical  procedure  was 
found  to  be  a  "quick  and  dirty"  method  that  could  be  useful  as  a 
check  after  a  reliability  demonstration  test  but  was  too  inexact 
and  subjective  for  final  validation  of  the  exponential 
assumption.  A  numerical  procedure  that  would  allow  a  formal  test 
of  hypothesis  was  sought. 

The  most  well-known  test  for  goodness  of  fit  is  the  chi-square 
test.  To  use  this  test  the  time  axis  is  divided  into  a  number  of 
intervals.  Each  of  the  k  intervals  forms  a  "class."  Denots  the 
class  boundaries  by 

ti  s 


The  failure  rate  parameter  is 
sample  mean: 

X 


ta  i 


(251) 


estimated  by  the  reciprocal  of  the 


(252) 
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On*  computes  the  statistic 


(253) 


where  o,  is  the  number  of  observed  interfailure  times  that  fall 
into  class  i  and  e,  is  the  expected  (theoretical)  number  of 
interfailure  times  that  would  fall  into  class  i.  The  expected 
number  of  inter failure  times  falling  into  class  i  is 

-  n[F(ci)-f(ci.l)]  (234) 

The  statistic  %2  is  distributed  as  chi-square  with  (k-1)  degrees 
of  freedom.  The  test  is  very  sensitive  to  the  number,  size,  and 
position  of  the  chosen  intervals.  The  sensitivity  problem  would 
only  go  away  with  larger  sample  sizes  (number  of  observed 
interfailure  times)  than  were  available. 

A  number  of  studies  [see  Moran  (1951),  Bartholomew  (1957)]  have 
shown  that  an  increasingly  popular  method  called  Bartlett's  test 
is  the  most  powerful  available  test  for  discriminating  among 
increasing,  constant,  and  decreasing  failure  rates.  The  test 
statistic  is 


(255) 


Under  the  null  hypothesis  of  exponentially  distributed 
interfailure  times,  the  statistic  Br  is  chi-square  distributed 
with  (n-1)  degrees  of  freedom.  A  two-tailed  test  is  used.  Let 
the  level  of  significance  be  denoted  a.  The  critical  values  for 
the  two-tailed  test  are 

Xi-«/a.n-i  and  x«/a,n-i  <ase> 

If  the  statistic  falls  between  these  two  values,  the  test  does 
not  contradict  the  null  hypothesis  that  the  exponential  model 
applies  to  the  failure  data.  Table  7-1  provides  some  percentage 
points  of  the  chi-square  distribution  [AMCP  (1968)]. 
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Table  7-1.  Percentage  Points  of  the  Chi-Square 
Distribution  for  a*. 01  Two-Tailed  Test 


Degrees  of  Freedom 

y  2 

2 

2 

.0100 

10.597 

3 

.0717 

12.838 

4 

.207 

14.860 

5 

.412 

16.750 

6 

.676 

18.548 

7 

.989 

20.278 

8 

1.344 

21.955 

9 

1.735 

23.589 

10 

2.156 

25.188 

11 

2.603 

26.757 

12 

3.074 

28.300 

13 

3.565 

29.819 

14 

4.075 

31.319 

15 

4.601 

32.801 

16 

5.142 

34.267 

17 

5.679 

35.718 
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Table  7-1  (con't) .  Percentage  Points  of  the  Chi-Square 


Distribution  for  a*. 01  Two-TaJ 

.led  Test 

Degrees  of  Freedom 

<y2 

X  005 

Y2 

18 

6. 265 

37.156 

19 

6.844 

38.582 

20 

7.434 

39.997 

21 

8.034 

41.401 

22 

8.643 

42.796 

23 

9.260 

44.181 

24 

9.886 

45.558 

25 

10.520 

46.928 

26 

11.160 

48.290 

27 

11.808 

49.645 

28 

12.461 

50.993 

29 

13.121 

52.336 

30 

13.787 

53.672 

35 

17.156 

60.304 

40 

20.674 

66.792 

It  is  important  that  the  data  be  from  one  series  of  runs  and  not 
"pooled,"  because  this  could  cause  misleading  results  [Kapur  and 
Lamberson  (1977)  Cox  and  Smith  (1954)]. 


Failure  data  sat  A-l  consists  of  the  following  collection  of  14 
interfailure  times  (in  CPU  minutes):  136,  304,  231,  13,  136, 
145,  13,  306,  231,  47,  462,  326,  33,  and  142.  The  computed  value 
of  Bartlett's  statistic  is  17.526.  Since  at  13  degrees  of 
freedom,  lies  between  x2  095«3.565  and  X2.oo5“29,819'  the  null 
hypothesis  that  the  interfailure  times  are  exponentially 
distributed  is  accepted. 


Failure  Data  Set  A-2  consists  of  the  following  collection  of  17 
interfailure  times  (in  CPU  minutes):  9,  110,  9,  32,  3,  19,  18, 
12,  65,  58,  9,  16,  20,  42,  2,  15,  and  86.  The  computed  value  of 
Bartlett's  statistic  is  14.519.  Since  at  15  degrees  of  freedom, 
14.159  lies  between  x2. 995*5. 142  and  X2  oos“34 •  267,  the  null 


hypothesis  that  the  in&erfailure 
distributed  is  accepted. 


^  .  .003 

times 


are  exponentially 
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Failure  Data  Set  A-3  consists  of  the  following  collection  of  17 
interfailure  tines  (in  CPU  minutes):  22,  10,  9,  130,  69,  26,  15, 
34,  55,  31,  139,  181,  41,  259,  43,  201,  and  261.  The  computed 
value  of  Bartlett's  statistic  is  15.532.  Since  at  17  degrees  of 
freedom,  15.532  lies  between  x2‘W5"5.697  and  ^20. 005.35 . 718 ,  the 
null  hypothesis  that  the  interfailure  times  are  exponentially 
distributed  is  accepted. 


Failure  Data  Set  A-4  consists  of  the.  following  collection  of  9 
interfailure  times  (in  CPU  minutes):  1,  63,  107,  23,  71,  62,  212, 
39,  and  246.  The  computed  value  of  Bartlett's  statistic  is 
9.763.  Since  at  8  degrees  of  freedom,  9.763  lies  between 
X2W5“1.344  and  %  pos "34.267,  the  null  hypothesis  that  interfailure 
times  are  exponentially  distributed  is  accepted. 


7.2.2  ,CS  Yttlffl  JTtBt ) 

The  Basic  Execution  Time  Model  [Musa  et  al.  (1987)]  is  based  on 
the  nonhomogeneous  Poisson  process  (NHPP) .  An  NHPP  is  completely 
characterized  by  its  intensity  function  (failure  rate)  i(r)  or 
its  mean  value  function  p ( t ) .  The  intensity  function  of  the 
Basic  Execution  Time  Model  is  not  a  new  innovation;  Parzen  (1962) 
described  the  intensity  function  form 

l(t)  p  a  exp(-bt]  (257) 

where  a  and  b  are  empiric-1 Ty  determined  positive  constants,  as 
"frequently  chosen"  for  events  that  hava  a  decreasing  intensity 
of  occurrence.  In  the  case  of  the  Basic  Execution  Time  Model, 
the  constant  a  is  interpreted  the  initial  failure  rate  and  b 
is  interpreted  as  p,  the  decrement  of  failure  rate  per  mean 
failure  experienced. 


As  discussed  in  the  chapter  on  Demonstration  Testing,  an  NHPP  can 
be  turned  into  a  homogeneous  Poisson  process  via  a  transformation 
of  the  time  scale.  Each  observed  failure  time  epoch  r,  is 
transformed  to 

Ui  -  M*i)  (258) 


Now  that  a  homogeneous  Poisson  process  is  obtained,  Bartlett's 
test  can  be  performed  as  above. 

To  determine  the  mean  value  function  M(r),  it  is  necessary  to 
obtain  estimates  for  the  Basic  Execution  Time  Model  parameters  p 
and  v0.  Then  is  the  transformed  failure  time  epoch  is 

■  |i(ti)  -  Vgd-expt-PTj  ) ,  i >1,2 (259) 
The  interfailure  times  are  calculated  as 
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Ui  -  Ui  -  Ut.lf  1*2 ,3 , k 


(260) 


with 


ui  -  uj. 


(261) 


For  failure  data  aat  B-l  (Tabla  7-2),  Bp-19. 056111,  whioh  lias 
batwaan  tha  critical  valuaa  x2  999*8 .142  and  x2  qq«*34.267.  80  tor 
B-l,  tha  null  hypothasia  is  accaptad.  For  failura  data  aat  B-2 
(Tabla  7-3),  Br»3.9958,  which  liaa  batwaan  tha  critical  valuaa 
X2  095*1.344  and  x2  005  *21.955.  So  for  B-2,  tha  null  hypothaais  ia 
accaptad.  For  failura  data  aat  B-3  (Tabla  7-4).  Br«14. 828652, 
which  liaa  batwaan  tha  critical  valuaa  x2  995*3.074  and 
X2  005*28. 300.  So  for  B-3,  tha  null  hypothaaia  ia  accaptad.  For 
fa'ilura  data  aat  B-4  (Tabla  7-5),  Bp«16. 073745,  which  liaa 

batwaan  tha  critical  valuaa  X2e«*4.60l  and  X2  005-32.801.  So  for 
B-4,  tha  null  hypothaaia  ia  accaptad. 


Tabla  7-2.  Data  Sat  B-l 


B-l  Failura  # 

Tina  (CPU  min) 

Tima  (M) 

Intarfailura 

1 

11 

0.359 

2 

24 

0.773 

0.414 

3 

31 

0.992 

0.219 

4 

48 

1.513 

0.521 

5 

50 

1.573 

0.060 

6 

89 

2.705 

1.132 

7 

98 

2.954 

0.249 

8 

128 

3.758 

0.804 

9 

136 

3.965 

0.207 

10 

156 

4.469 

0.504 

11 

178 

0.003 

0.534 

12 

249 

6.586 

1.583 

13 

574 

11.702 

5.116 

14 

587 

11.850 

0.148 

15 

1023 

15.230 

3.380 

16 

1391 

16.587 

1.350 

17 

2240 

17.701 

1.114 

18 

2449 

17.796 

0.095 
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Table  7-3.  Data  Set  B-2 


B-2  Failure  # 

Time  (CPU  min) 

Time  (u) 

Interfailure 

1 

35 

2.878 

2 

37 

3.018 

0.140 

3 

55 

4.171 

1.153 

4 

58 

4.346 

0.175 

5 

71 

5.005 

0.659 

6 

85 

5.735 

0.730 

7 

93 

6.087 

0.352 

8 

127 

7.341 

1.254 

9 

135 

7.586 

0.245 

10 

170 

8.480 

0.894 

Table  7-4.  Data  Set  B-3 


B-3  Failure  # 

Time  (CPU  min) 

Time  (m) 

Interfailure 

1 

4 

0.355 

2 

19 

1.608 

1.253 

3 

25 

2.077 

0.469 

4 

55 

4.167 

2.090 

5 

57 

4.292 

0.125 

6 

63 

4.659 

0.367 

7 

93 

6.296 

1.637 

8 

130 

7.926 

1.630 

9 

135 

8.118 

0. 192 

10 

138 

8.230 

0.112 

11 

139 

8.268 

0.038 

12 

166 

9.180 

0.912 

13 

441 

13.176 

3.996 

14 

576 

13.654 

0.478 
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Table  7-5.  Data  Set  B-4 

nMBMK^BaMaBBnKB 


B-4  Failure  # 

Time  (CPU  min) 

Interfailure 

1 

11 

0.550 

2 

18 

0.892 

0.342 

3 

50 

2.381 

1.489 

4 

71 

3.295 

0.914 

5 

80 

3.672 

0.377 

6 

107 

4.752 

1.080 

7 

144 

6.117 

1.365 

8 

153 

6.430 

0.313 

9 

205 

8.107 

1.677 

10 

342 

11.597 

3.490 

11 

350 

11.766 

0.169 

12 

351 

11.786 

0.020 

13 

358 

11.931 

0.145 

14 

364 

12.052 

0.121 

15 

494 

14.284 

2.232 

16 

558 

15.140 

0.856 

17 

620 

15.847 

0.707 

Next,  growth  techniques  were  applied  to  ongoing  development 
projects.  Demonstration  testing  techniques  served  to  validate 
the  results  of  the  growth  testing.  The  team  chose  a  "short  run 
high  risk  teat  plan"  because  the  test  duration  is  short  and  so 
could  be  repeated  many  times.  The  Growth  Data  Setts  are  shown  in 
Table  7-6.  The  failure  rate  l(r,0)  is  assigned  to  the  lower  test 
failure  rate  10.  The  decision  risks  are  30%.  The  demonstration 
test  test  has  a  discrimination  ration  of  3,  so  is  set  to  10  x 
3.  The  test  duration  is  1.1  x  ni"1 .  The  test  rejects  if  1  or 
more  failures  occur.  The  test  accepts  only  if  0  failures  occur. 
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Table  7-6.  Growth  Data  Sets 


993 


3430 

3452 

3520 


587 

604 


For  Data  Set  04 ,  the  growth  model  provided  a  failure  rate 
estimation,  at  the  3Gth  failure,  of  0.0011365.  The  test  duration 
is  computed  to  he  28.24.  The  demonstration  test  was  repeated  13 
times  The  software  passed  9  tests  and  failed  4  tests.  If  the 
true  failure  rate  is  that  given  by  the  growth  model  and  set  to 
1„,  then  the  demonstration  test  should  falsely  reject  the 
software  30%  of  the  time.  Since  this  in  fact  happened 

•  0.3076923  <2«2) 


fraction  of  the  time,  the  agreement  with  this  data  set  is  quite 
close.  On  the  other  data  sets,  the  raw  growth  model  provided  an 
overly  optimistic  failure  rate,  with  the  result  that  the  software 
could  not  pass  the  demonstration  test.  When  the  model  was 
recalibrated  (as  described  in  Section  5) ,  the  optimism  was 
corrected.  The  recalibration  technique,  however,  does  not  alter 
the  failure  rate,  only  the  probabilities  of  failure.  A  future 
topic  of  research  would  be  to  set  i0  based  on  the  results  of  a 
recalibrated  model. 

7.2.3  validation.,  of,  Salty ara  .£aiIurfl„.Rato.  Baas&ialflflL Moncla 

In  response  to  SOW  paragraph  4. 1.6. 4,  the  software  reliability 
prediction  techniques  were  applied  to  actual  projects.  The 
results  wers  tne  compared  to  the  actual  reliability  measured 
during  growth  testing. 


7. 2. 3.1  Approach 


In  Appendix  E,  several  regresssion  models  that  describe  the 
relationship  between  software  failure  rate  (V)  and  software 
characteristics  (x1,...,x«)  are  described.  In  this  section,  only 
the  linear  regression  models  that  are  developed  for  each  phase 
(Including  the  model  that  was  chosen  for  the  previous  phase)  of 
the  software  development  are  compared  to  one  another,  since  it 
was  discovered  that  the  nonlinear  regression  models  possess  the 
same  statistical  characteristics  as  those  of  the  linear  models. 
Finally,  only  one  regression  model  is  chosen  to  predict  the 
software  failure  rate  after  each  phase  of  the  software 
development. 

As  a  selection  criteria  for  the  prediction  model  for  each  phase, 
the  following  ground  rules  were  applied: 

1.  The  model  with  the  best  correlation  to  the  observed  software 
failure  rate  was  chosen  whenever  there  were  any  significant 
differences  in  correlation  between  the’ two  models. 
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2.  When  two  models  produced  essentially  identical  correlation 
between  the  predicted  software  failure  rate  and  the  failure 
rate  observed  on  the  projects,  the  model  which  accounted  for 
the  largest  number  of  metrics  (most  recent  information)  was 
selected. 

The  need  to  select  the  model  which  provided  a  software  failure 
rate  with  the  best  statistical  correlation  to  the  observed 
failure  rate  is  obvious.  'The  study  team's  preference  for  a 
prediction  model  which  used  the  largest  number  of  metrics  is 
based  on  two  areas  of  concern  with  respect  to  the  data  collected 
during  the  study  period.  The  data  collected  to  support 
development  and  validation  of  the  prediction  models  was  all 
collected  after  all  system  integration  testing  on  the  program  was 
complete.  In  most  cases,  the  software  was  in  use  at  the  customers 
facility.  As  a  result,  the  metric  data  collected  on  the  projects 
was  essentially  complete.  Thus,  the  data  available  to  the  study 
team  was  much  more  accurate  and  complete  than  the  data  which 
would  normally  be  available  at  the  end  of  a  given  program  phase 
(eg  requirements  analysis) .  This  may  have  resulted  in  much  better 
accuracy  for  the  early  program  phase  models  than  would  normally 
be  expected.  Also,  the  metric  data  collected  during  the  project 
was  limited  to  one  company  (Huqhes  Aircraft)  and  thus  potentially 
valuable  variations  in  values  of  the  metric  data  which  would 
occur  between  companies  was  cmmitted,  possibly  yielding  better 
stability  and  accuracy  for  the  early  prediction  models  than  would 
ordinarily  result . 


• 

Two  more  observations,  besides  the  nine  observations  that  were 
used  to  derive  the  regression  models  in  Appendix  E,  were 
available  to  validate  these  models.  These  two  observations  are 
identified  as  obs.10  And  obs.il. 


The  following  statistics  will  be  used  for  comparison: 

-  Relative  error  of  the  predicted  failure  rate  of  each 
observation  (obs.10  or  obs.ll)  in  ths  regression  model 

-  R2 

-  Residuals  plots 

-  Prob  >  P 

-  Prob  >  | T| 

-  Standard  errors  of  the  parameter  estimates 

-  The  length  of  the  95%  confidence  limits  for  a  mean 
predicted  values  for  each  observation  (obs.10  or  obs.ll),  with 
the  lover  limit  being  0,  if  its  calculated  value  is  negative. 
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The  linear  regression  models  which  are  derived  in  Appendix  E  are 
of  the  form 

Y  -  0O  +  0TX  +  e  (263) 

where 

Y :  software  failure  rate 

0. :  the  y- intercept 

0r:  an  l  x  a  vector  of  real  numbers 

it  in  1  x  b  vector  of  independent  variables 

e:  error  variable  of  the  regression  model 

m  S  24 

these  regression  models  are  formulated  based  on  the  following 
assumptions : 

i)  E(e)  -  0  (i.e.,  the  expected  value  of  e  is  0) 

ii)  The  variance  of  t  is  the  same  for  all  value  of  x 

iii)  The  values  of  e  are  independent 

iv)  e  is  a  normally  distributed  random  variable. 

The  validity  of  these  assumptions  in  each  regression  model  is 
determined  based  on  its  residuals  plot. 


7. 2. 3. 2  Biaul&a 


The  regression  models  which  were  developed  for  each  phase  of  the 
software  development  (Appendix  E) ,  were  compared  to  one  another. 
Then,  the  chosen  model  was  compared  to  the  model  that  had  been 
selected  for  the  previous  phase.  Cne  regression  model  was 
selected  for  each  software  development  phase  to  describe  the 
relationship  between  software  failure  rate  and  software 
characteristics . 


(i)  Biaalcaninfca  Analysis  Phan 


Three  regression  models  were  formulated  for  the  requirements 
analysis  phase  by  using  the  least  squares  method,  stepwise 
regression  and  principal  components  analysis. 

Only  five  software  characteristics  can  be  identified  after  the 
Requirements  Analysis  phase.  Therefore,  only  five  independent 
variables  are  involved  in  these  regression  models. 

A  complete  output  of  these  models  is  provided  at  the  end  of 
Appendix  E.  A  summary  of  these  regression  models  (least  squares 
model,  stepwise  model,  and  principal  components  model)  is  given 
below. 
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Y  -  5.13  +  15.72X,  -  0.17x2  +  0.52x3  +  0.39x4  -  0.23x5 

Y  is  restricted  to  be  non-negative. 

-  atiBaim  modal ,  (M2,)  i 

Y  ■  11.46  +  12 . 58x1  -  0.08x2  (Y  is  restricted  to  be  non¬ 
negative) 


Principal  components  regression  mode3^M3V;_ 

Y  -  18.04  +  0.05  P,  (Y  is  restricted  to  be  non-negative) 
P1  -  0 •  009x1  +  0 . 99Xj  +  0.10x3  -  0.0001X4  f  0.0005X, 


Table  7-7  provides  a  summary  of  the  values  of  statistical 
characteristics  of  models  Ml ,  M2  and  M3. 

Although  the  value  of  R2,  in  model  Ml,  is  high  (>  0.80,  i.e., 
more  than  80%  of  the  total  variation  in  the  failure  rates  sampled 
is  explained  by  the  regression  model  Ml) ,  the  value  of  Prob  >  P  » 
0.18,  which  is  bigger  than  0.15  (the  predetermined  significance 
level) ,  suggested  that  the  null  hypothesis  in  the  hypothesis  test 
(I),  described  below,  should  be  accepted.  Thus,  Ml  is  not  an 
adequate  model  to  describe  the  relationship  between  Y  and  the 
independent  variables  x1 , . . . ,  x, . 

Hypothesis  test  (I) : 

H0:  all  coefficients  of  the  independent  variables  in  the 
regression  model  are  0. 

Ht:  there  exists  at  least  one  coefficient,  that  is  not 
equal  to  0,  of  an  independent  variable. 


Other  reasons  to  reject  Ml  are  that  the  values  of  the  relative 
error  of  the  predicted  Y  in  obs.10  are  too  high,  and  the  length 
of  the  95%  confidence  limits  of  the  predicted  value  for  obs.10  is 
extremely  large  (»  592) . 
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Table  7-7.  Summary  of  Statistical  Values  of  Models  Ml,  M2  and  M3 

Modei™^ "™""1 


Prob  >  F 


Standard 

Error 


sw 

Failure 

Rate 


y- intercept 


Predicted 


Actual 


Abs. 

Error 


Rel. 

Error 


95% 
Conf . 
Limits 


Obs. 

10 

Obs. 

11 

Obs. 

10 

Obs. 

11 

Obs. 

10 

Obs. 

11 

Obs. 

10 

Obs. 

11 

Obs. 

10 

Obs. 

11 

Ml 

M2 

M3 

.8419 

.8137 

.4064 

.1839 

.0065 

.0648 

13.38 

4.35 

6.54 

8.32 

3.48 

0.02 

0.14 

0.04 

n/a 

0.85 

n/a 

n/a 

2.54 

n/a 

n/a 

2.44 

n/a 

n/a 

0.72 

0.04 

0.03 

0.16 

0.01 

0.06 

0.29 

0.08 

n/a 

0.58 

n/a 

n/a 

0.89 

n/a 

n/a 

0.93 

n/a 

n/a 

234.4 

182.9 

18.5 

2’  .97 

24.98 

21.03 

36 

36 

36 

15 

15 

15 

198.4 

146.9 

17.5 

6.97 

9.98 

6.03 

■sal 

4.08 

0.486 

0.46 

0.66 

0.402 

68-297 

0-62.9 

0-72.7 

0-52.7 

0-65.2 

Model  M2  is  better  than  model  Ml,  since  it  yields  smaller  values 
of  relative  errors  and  smaller  lengths  of  the  95%  confidence 
limits  for  the  predicted  values  of  obs.10  and  obs.ll  compared  to 
those  of  Ml.  But,  because  x,,,...^,  are  highly  correlated, 
stepwise  regression  analysis  is  not  reliable  for  producing  a 
valid  regression  model.  In  fact,  even  though,  M2  is  better  than 
Ml,  the  predicted  failure  rates  of  obs.10  are  still  too  high 
compared  to  the  actual  vaJue,  and  the  95%  confidence  interval  of 
the  predicted  value  for  obs.10  does  not  contain  the  actual 
failure  rate. 

Compared  to  models  Ml  and  M2,  model  M3  is  preferable,  even  though 
the  value  of  R2  of  M3  is  much  smaller  than  that  of  Ml  and  M2  (in 
modal  M3,  only  41%  of  the  total  variation  of  the  failure  rate 
samples  is  explained  by  the  regression  model) .  The  p-value  of 
hypothesis  test  (I)  and  the  p-value  of  hypothesis  test  (II) 
(which  is  defined  on  page  E-21)  of  each  parameter  estimate  are 
smaller  than  0.15  (the  predetermined  significance  level).  This 
means  P,  (independent  variable)  is  relevant  in  predicting  the 
software  failure  rate  (the  dependent  variable)  in  M3. 

Other  reasons  for  choosing  M3,  instead  of  Ml  and  M2,  are  that  the 
relative  errors  of  the  predicted  failure  rate  in  obs.10  and 
obs.ll  are  reduced  substantially  from  those  of  Ml  and  M2,  and 
that  these  relative  errors  are  similar  to  each  other  (i.e.,  M3 
yields  consistent  errors  in  predicting  failure  rates  for 
different  software).  Furthermore,  based  on  the  residuals  plot  of 
M3  (P1  versus  residuals),  on  page  E-36,  it  is  observed  that  all 
assumptions  listed  above  hold  (i.e.,  the  residuals  are  of 
constant  variance,  and  independent) .  Therefore,  M3  is  a  good 
model  to  use  for  predicting  software  failure  rate  based  on  the 
software  characteristics  that  can  be  obtained  after  the 
requirements  analysis  phase. 

(2)  Preliminary  Design  Phase 

Similar  to  the  requirements  analysis  phase,  there  are  three 
regression  models  that  are  developed  for  use  to  predict  the 
software  failure  rate  after  the  preliminary  design  phase.  These 
models  were  formulated  by  using  stepwise  regression  and  principal 
components  analysis.  Specifically,  two  of  the  model a  are 
developed  by  using  principal  components  analysis.  All  of  these 
models  can  be  summarized  as  below. 


-  s&BByiftf,  mgflfll  LMIjl 

Y  -  11.92  +  14.12X,  -  0.14x9  (V  is  restricted  to  be  non¬ 
negative) 
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Y  -  17.90  +  0.04  P,  4  0.01  P2  (Y  is  restricted  to  be  non 
negative) 


P.  »  0. 007x.  4  0. 796x2  4  O.O8X3  -  0.0003x4  4  0 . 0003X5  4 

i.00009X4  +-0.0043x7  4  O.OlOXg  4  0.6X9  4  0.003X10 


P2  -  0.03X1  4-  0.45X2  4  0 . 56Xj  4  0.11x4  4  0.08X,  -  0 . 005x6  4 
0.07X7  4  0.01X3  -  O.67X9  4  0 . 08x10 


-  Principal  Components  regression  model  M6i 
Y  ■  17.90  4  0.04  P,  (Y  Is  restricted  to  be  non-negative) 

P1  :  same  as  above 

Table  7-8  provides  the  summary  of  the  statistical  values 
associated  with  the  above  models. 


Model  M6  is  derived  in  order  to  eliminate  P.  in  M5,  since  the  p- 
value  of  the  hypothesis  test  (II)  for  variable  P2  is  too  high 
compared  to  0.15  (the  predetermined  significance  level). 
Therefore,  only  models  M4  and  M6  and  the  previously  chosen  model 
(M3)  will  be  compared  to  one  another. 


Model  M4  is  inferior  M6  in  describing  the  relationship  between 
software  failure  rate  (the  dependent  variable)  and  software 
characteristics  (the  independent  variables)  which  can  be 
identified  after  the  preliminary  design  phase.  The  reason  for 
this  is  that,  even  though  the  value  of  R2  is  high  and  the  p- 
values  of  hypothesis  tests  (I)  and  (II)  are  low  for  M4,  the 
absolute  error  (hence,  the  relative  error)  of  the  predicted 
software  failure  rate  of  obs.10  is  quite  larger  Also,  the  95% 
confidence  limits  interval  of  the  predicted  failure  rate  in 
obs.10  does  not  contain  the  actual  value,  and  the  length  of  the 
entire  interval  is  too  large.  Again,  it  is  not  surprising  that 
the  stepwise  regression  model  is  not  good  in  predicting  software 
failure  rate  in  this  case,  since  there  is  a  high  correlation 
between  the  independent  variables  xu...,x10. 
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Table  7-8.  Summary  of  Statistical  Values  of  Models  M4,  M5  and 

M6 


Model 

M4 

M5 

M6 

R2 

0.6584 

0.4016 

0.40 

Prob  >  F 

0.0024 

0.21 

0.07 

Standard 

Error 

y-intercept 

3.64 

7.13 

6.60 

*i 

3.07 

0.02 

0.02 

*2 

0.05  . 

0.42 

0.03 

Prob  > 

1*1 

y- intercept 

0.02 

0.05 

0.07 

*i 

0.003 

0.09 

18.66 

*2 

0.03 

0.98 

21.58 

sw 

Failure 

Rate 

Predicted 

Obs.  10 

202.2 

18.75 

18.66 

Obs.  11 

22.68 

21.39 

21.58 

Actual 

Obs.  .10 

36 

36 

36 

Obs.  11 

15 

15 

15 

Abs. 

Error 

Obs.  10 

166.2 

17.25 

17.34 

Obs.  11 

7.68 

6.39 

6.58 

Rel. 

Error 

Obs.  10 

4.62 

0.48 

0.48 

Obs.  11 

0.51 

0.43 

0.44 

95%  Conf. 
Limits 

Obs.  10 

101-303 

0-69.58 

0-63.3 

0-46.14 

0-74.52 

0-65.9 
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Models  M3  and  M6  were  compared  to  each  other.  Notice  that 
although  model  M6  has  similar  residuals  plot  as  that  of  M3,  model 
M6  has  better  values  for  all  statistics  compared  to  M3. 
Therefore,  M6  was  chosen  to  be  the  regression  model  that 
describes  the  relationship  between  software  failure  rate  and 
software  characteristics  that  are  available  after  the  preliminary 
design  phase. 

(3)  Detailed  Design  Phase 


A  summary  of  the  regression  models  that  were  developed  for  the 
detailed  design  phase  is  provided  below: 


-  Stepwise  -model . <M7)  t 


Y  -  28.97  +  15. 63x1  +  0.25x6  -  2.39XB  -  O.O8X9  -  1.2xn 

(Y  is  restricted  to  be  non-negative) 

-  Principal,  components  regression  model  (MB) : 


Y  -  36.62  +  0.13  P1  -  2.47  P2  -  0.64  P3  (Y  is  restricted  to  be 
non-negative) 


o.oobix6  + 

0.004x13  - 


■  0.007X.  +  0 . 8 Ox,  +  0.08x3  -  0.0003x4  +  0.0003X, 
0.004X7  +  0.01x8  +  0.6X,  +  O.OO3X10  +  O.llx,,  4-  0.002x12 
0.003xu‘ 


+ 

+ 


Pj  ■  0.01X,  +  0 .  40x*  +  0 . 49Xj  +  0 . 13X/  +  0.08x5  +  0.02x6  +  0.08x7 
+  O.OlXg  -  0 •  6X9  +  O.O9x10  -  0.26X,,  +  0.11x12  +  0.09x13  -  0.35XU 

P3  ■  -0.25X,  +  0.02xa  -  0 . 26x3  f  0.20x4  *•  0.07xs  +  0.63x6  + 
0.10X7  -  0 . 12xfl  +  O.OlX,  +  O.llX10  +  0.35X,,  +  0.11X12  -  0.10x13  - 
0.49XU 

Table  7-9  provides  a  summary  of  the  statistical  values  associated 
with  these  models  (M7  and  M8) . 


Similar  to  reasons  given  above  for  rejection  of  stepwise 
regression  model  M4,  stepwise  model  M7  should  also  be  discarded. 
Even  though  the  values  of  all  statistics  of  this  model  are 
reasonably  good,  when  it  is  used  to  predict  the  software  failure 
rates  of  obs.10,  the  absolute  error  (hence,  the  relative  error) 
of  the  predicted  value  is  too  high.  Also,  based  on  the  relative 
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Table  7-9.  Summary  of  Statistical  Values  of  Models  M7  and  K8 


95%  Conf 
Limits 


Obs.  11 


5.95-24.4 


0-62.1 


errors  of  the  predicted  values  of  obs.10  and  obs.ll,  it  can  be 
stated  that  M4  is  not  consistent  in  predicting  the  software 
failure  rate.  In  one  caae  (obs.ll),  it  yields  very  good 
prediction  and  95%  confidence  limits,  but  in  the  other  case 
(obs.10),  M4  predicts  the  value  of  the  software  failure  rate 
poorly  (the  predicted  value  is  491%  of  the  actual  valuei). 

Next,  model  M8  is  compared  to  model  M6,  which  was  chosen  as  the 
regression  model  after  the  preliminary  design  phase. 

M8  seems  to  yield  good  predicted  value  for  the  software  failure 
rate  in  the  case  of  obs.10,  and  the  value  of  R*  of  M8  is  high. 
Also,  the  value  of  Prob  >  F  is  much  smaller  than  that  of  M6.  But, 
MB  is  inconsistent  in  predicting  the  dependent  variable  (Y) .  For 
instance,  the  relative  error  of  obs.10  is  only  0.4,  but  the 
relative  error  of  obs.ll  is  0.79.  Also,  even  though  the  actual 
failure  rate  in  obs.10  is  twice  as  much  as  that  of  obs.ll,  the 
upper  bound  of  the  95%  confidence  limits  of  obs.10  is  smaller 
than  the  upper  bound  for  obs.ll.  Another  disturbing  factor  of 
model  M8  is  that  the  p-value  of  the  hypothesis  test  (II)  of  P2  is 
much  higher  than  0.15  (the  predetermined  significance  level). 
This  suggested  that  P2  is  not  relevant  in  predicting  software 
failure  rate  using  M8. 

Therefore,  once  again,  M6  is  chosen  to  be  the  regression  model 
that  can  be  used  to  predict  the  software  failure  rate  based  on 
the  software  characteristics  available  after  the  detailed  design 
phase . 

(4)  flail  ind  unit  Tut  Phan 

It  aan  be  safely  stated  that  stepwise  regression  models  should 
not  be  considered  in  companion  with  principal  components  models, 
since  there  is  substantial  evidence  that  stepwise  analysis  does 
not  work  well  with  correlated  independent  variables,  whereas 
principal  components  analysis  does.  Thus,  in  this  section,  only 
regression  models  that  are  developed  using  principal  components 
analysis  are  examined. 

Two  principal -components  regression  models  were  formed.  These 
models  can  be  summarized  below. 
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Y  ■  38.03 


(Y  is  restricted  to  be  non-negativa) 


-  2.59  P, 


-  0.0001X,  -  0  00002X,  - 

-  0.00002xa  -  0 . 0006X,,  * 

-  0  •  OOOOlX.*  +  0  •  005X1b  - 

0. 000004x22  -  O.OOOOlXjj  - 


-  RaqraiilQn -Modal. iMi 


P.  *  -O.OOOOlX.  -  0 . 0009x2 

0.00002X,  -  0.00003Xx  -  0.000017X, 

0 .  OOO2x10  +  O.OOOOlX..  -  0 . 00002x^2 

0 .  OOOlX.n  -  0 .  OOOlXjn  -  O'.OOXj.  + 

0.00002x24  -  0.0001x25 


Y  -  17.88  +  0.04  PA  (Y  is  restricted  to  be  non-negative) 

P.  *  0.007X,  +  0.80x2  +  O.O8X3  +  O.OlXj  +  0 . 6X9  +  0.008X33 
+  0.0.3x2J 

Table  7-10  provides  the  summary  of  the  statistical  values 
associated  with  these  models  (M9  and  M10) . 

It  is  obvious  that  the  regression  model  M9  should  be  rejected. 
Since  both  p-values  of  hypothesis  tests  (I)  and  (II)  (of  the 
independent  variable  P,)  are  much  higher  than  0.15  (the 
predetermined  significance  level) ,  P1  is  not  relevant  in 
predicting  software  failure  rate  using  model  M9.  But,  if  P,  were 
eliminated  from  the  model,  a  new  model  cannot  be  formed.  Another 
reason  to  reject  model  M9  is  that  its  residuals  plot  indicates 
that  assumption  (ii)  is  violated.  A  surprising  element  was 
noticed  in  model  M9:  the  absolute  errors  (hence,  the  relative 
errors)  of  the  predicted  failure  rates  in  both  obs.10  and  obs.ll 
are  much  smaller  than  those  of  any  of  the  previous  models. 

When  model  M10  was  compared  to  model  M6  (which  was  chosen  after 
the  detailed  design  phase) ,  it  was  noticed  that  all  values  of  the 
statistics  obtained  for  M6  and  M10  are  vary  similar  to  each 
other,  including  the  residuals  plots.  Therefore,  theoret \eally, 
either  M6  or  M10  can  be  used  to  predict  the  software  failure  rate 
after  the  code  and  unit  test  phase. 

Model  M10  is  recommended  for  used  in  predicting  software  failure 
rates  after  code  and  unit  test  because  M10  includes  a  larger  set 
of  the  independent  variables  than  M6  and  thus  may  be  less 
succeptable  to  any  inadvertent  result  variations  caused  by  the 
data  collection  limitations  of  the  study  project. 


Table  7-10.  Summary  of  Statistical  Values  of  Models  M9  and 

M10 


Model 


M9 

M10 

R*  • 

0.0748 

0.4012 

Prob  >  F 

0.4764 

0.0670 

Standard 

Error 

y- intercept 

19.72 

6.61 

xi 

3.43 

0.02 

Prob  > 

y- intercept 

0.0952 

0.0304 

|T| 

xi 

0.4764 

0.0670 

Predicted 

Obs.  10 

22.72 

18.68 

Obs.  11 

11.20 

21.59 

Actual 

Obfl.  10 

36 

36 

Obe.  11 

15 

15 

SW  Failure 
Rate 

Abs. 

Error 

Obs.  10 

13.28 

17.32 

Obs.  11 

3.80 

6.59 

Rel. 

Obs.  10 

0.37 

0.48 

Error 

Obs.  ll 

0.25 

0.44 

95%  Conf. 
Limits 

Obs.  10 

0-77.9 

0-63.3 

Obs.  11 

0-79.8 

0-65.9 
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e.  CONCLUSIONS  AND  RECOMMENDATIONS 
8.1.  Conclusions 

This  st  ;dy  has  developed  techniques  for  reliability  prediction, 
allocation,  growth  testing,  and  demonstration  testing  of  combined 
hardware/softvare  systems.  The  software  reliability  techniques 
are  compatible  with  existing  hardware  reliability  standards  and 
procedures  and  are  aligned  with  DOD-STD-2167A. 

The  main  findings  of  the  research  are: 

A.  Software  whose  code  is  frozen,  and  that  is  being  subjected  to 
a  stationary  operational  profile,  can  be  reasonably  modeled  as 
having  a  constant  failure  rate  (see  Appendix  B)  .  Before  this 
result  was  obtained,  researchers  in  software  reliability 
justified  the  constant  failure  rate  model  primarily  on  intuitive 
considerations.  The  work  dene  here  puts  the  model  on  a 
theoretical  foundation.  Because  hardware  components  during  their 
useful  life  period  and  maintained  electronic  systems  are 
conventionally  modeled  by  a  constant  failure  rate,  this  means 
that  certain  hardware  reliability  techniques  have  analogies  in 
software.  Furthermore,  hardware  and  software  failures  can  be 
combined  with  each  other.  For  example,  if  the  hardware  has  a 
certain  failure  rate  and  operates  concurrently  and  in  series  with 
the  software ,  the  hardware  and  software  failure  rates  can  be 
added  to  obtain  the  overall  failure  rate. 

B.  Software  reliability  can  be  predicted  based  on  measurable 
characteristics  of  the  software  development  process  and  work 
product's.  Prediction  models  based  on  metrics  initially  available 
early  on  in  the  software  life  cycle  appear  to  have  about  the  same 
prediction  power  as  those  models  that  add  metrics  available 
lates ,  when  the  early  metrics  are  revised  to  reflect  updated  work 
products . 

C.  A  program's  failure  rate  is  related  to  the  program's  fault 
hazard  rate  profile.  Previously,  the  hazard  rate  profile  could 
only  be  determined  by  "fault  seeding"  or  by  retrospective  failure 
analysis.  Neither  method  is  practical.  The  study  developed  a 
way  of  determining  a  program's  hazard  rate  profile,  under  a 
particular  operational  profile,  by  adding  randomly  placed 
counters  to  the  code.  These  counters  provide  information  about 
the  frequency  with  which  potential  faults  are  encountered. 

0.  Markov  modeling  can  be  employed  for  the  modeling  and 
allocation  of  hardware/software  systems  that  involve 
hardware/software  repair  and  hardware  redundancy.  Modeling  and 
allocation  of  software  under  different  reliability  topologies  and 
timing  configure cions  c^n  generally  be  handled  by  closed-form 
expressions. 
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E.  Software  reliability  growth,  from  the  start  of  system  test, 
can  be  modeled  by  the  Basic  Execution  Time  Model.  If  the  number 
of  observations  is  small  ( <3 0 )  or  when  the  faults  in  a  program 
have  widely  different  hazard  rates,  then  the  model  may  falter. 
In  both  cases,  the  reliability  estimations  and  projections  can  be 
improved  through  numerical  recalibration.  A  drawback  to 
recalibration  is  that  it  currently  works  by  altering  the 
cumulative  distribution  function  (time-dependent  failure 
probabilities) ,  rather  than  by  altering  the  failure  rate.  An 
estimate  of  the  failure  rate  is  useful  in  designing  demonstration 
test  plans. 

During  growth  testing,  the  operational  profile  must  be  kept 
stationary  and  emulate  field  use.  Testers  must  record  failure 
times  (or,  equivalently,  inter failure  times)  in  terms  of 
execution  time.  This  "statistical K  approach  to  testing  is  an 
efficient  way  of  uncovering  those  faults  that  contribute  the  most 
to  the  overall  program  failure  rate.  Current  testing  practice  is 
either  to  stress  the  software  by  choosing  inputs— such  as 
boundary  values— ‘ that  are  likely  to  trigger  a  failure,  or  to 
attempt  to  "cover"  tha  input  space  or  code  in  some  way. 

F.  Three  types  cf  software  reliability  demonstration  test  have 
been  developed:  fixed-length  tests,  failure-free  execution 
interval  tests,  and  sequential  tests.  In  addition,  a  method  was 
presented  for  performing  a  demonstration  test  concurrently  with 
growth  testing. 

G.  Both  growth  testing  and  demonstration  testing  can  be 
accelerated  by  simultaneous  execution  of  multiple  copies  of  the 
program.  When  a  failure  occurs,,  ths  time  recorded  ie  the  total 
execution  time  accumulated  on  all  copies.  When  the  code  is 
modified,  it  must  be  modified  on  all  versions.  The  test  cases 
for  each  copy  must  be  selected  i independently  according  to  the 
same  operational  cofile. 

H.  The  random  selection  of  test  inputs  from  a  given  operational 
profile  can  be  performed  or  automated  based  on  the  fact  that  the 
operational  profile  induces  a  partition  of  the  real  interval 
(0,1).  Each  input  state,  or  class  of  input  states,  is  associated 
with  a  subinterval  of  (0,1)  whose  length  is  its  probability  of 
occurrence.  Testing  efficiency  can  be  improved  through  the  use 
of  equivalence  partitioning. 


8.2.  Recommendations 

Future  research  is  recommended  to  address  the  following  areas: 

A.  The  prediction  models  have  been  validated  in  one  corporation 
only.  The  product/process  metrics  and  failure  data  should  be 
collected  from  projects  in  many  organizations  so  that  the  model 
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can  be  validated  more  universally.  Additional  metrics  should  be 
hypothesized  and  collected,  to  find  more  that  are  useful  for 
predicting  software  reliability. 

D.  The  values  of  the  many  product/process  metrics  will  change  as 
early  work  products  are  updated.  The  time  or  phase  at  which  the 
value  of  a  metric  is  sampled  should  be  made  a  part  of  the 
regression  analysis. 

C.  The  current  recalibration  technique  alters  the  failure 
probabilities.  A  method  should  be  developed  for  recalibrating  of 
the  failure  rate,  altering  the  failure  rate  from  one  constant 
value  to  another.  One  approach  is  to  fit,  in  a  least-sguares 
sense,  the  recalibrated  cumulative  distribution  function  to  that 
of  ths  exponential  distribution,  the  constant  rats  paramatar 
traated  an  as  unknown. 

D.  Theory  and  techniques  need  to  bs  developed  for  quantifying 
tha  benefits  ?.nd  costs  of  multiversion  programming.  As  a 
prerequisite,  techniques  should  bs  developed  for  injecting 
diversity  into  software  in  such  a  way  as  to  deterministically 
increase  ths  reliability.  Currently,  there  is  no  way  of  knowing 
if  the  expense  of  creating  multiple  versions  is  cost  affective 
compared  to  additional  tasting  and  review. 

E.  An  automated  test  generator  program  should  be  developed  that 
uses  information  on  the  operational  profile,  and  the  data  types 
and  ranges  of  the  input  variables. 

F.  The  effect  of  software  engineering  technologies  on  software 
reliability  should  be  quantified  through  a  series  of  controlled 
experiments . 

G.  Automated  tools  should  bs  developed  for  accurately  recording 
cumulative  execution  time.  Currently,  teeters  use  calendar  time. 

H.  Research  should  be  performed  on  designing  software  to 
facilitate  reliability  prediction.  As  an  example,  consider  an 
abridged  operational  profile  that  specifies  the  relative 
frequency  of  the  usar-orisntsd  functions,  A,  B,  and  C.  Suppose 
that  the  software  is  designed  so  that  A,  B,  and  C  are  implemented 
in  separate  CSCs.  Let  cperational  profile  #1  spweify  that  tho 
relative  frequency  of  A  is  20%;  for  B,  30%;  and  for  C,  50%.  The 
failure  rate  measured  under  this  operational  profile  A  A.,  A,, 
and  Ac.  Now,  zuppose  that  operational  profile  #2  is  A,  30%;  B, 
40%;  and  C,  30%.  Without  performing  any  testing  under 
operational  profile  #2,  the  failure  rates  can  bs  predicted  by 
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(2<5) 


(266) 


I.  Appendix  B  shows  that  if  olustsrs  of  softvars  discrepancies 
(diffsrencss  bstwssn  actual  and  corrsct  values  of  output 
variables)  occur  according  to  a  Poisson  process,  then  the  number 
of  individual  discrepancies  is  governed  by  a  "stuttering  Poisson 
process."  Sines  for  many  programs  it  may  be  more  natural  or 
meaningful  to  count  discrepancies,  a  set  of  demonstration  teats 
should  be  developed  that  are  based  on  a  stuttering  Poisson  model. 

J.  More  research  should  be  done  regarding  the  effect  of 
complexity  on  software  reliability.  The  complexity  measures 
commonly  used  today  tend  to  correlate  with  size,  and  so  are  not 
of  use  in  prediction.  If  complexity  is  measured  not  as  intricacy 
of  structure  but  as  the  number  of  paths,  then  it  would  appear 
that  it  must  influence  fault  content,  since  a  program  is  only 
free  of  faults  to  the  extent  that  it  has  been  tested.  The 
structure  factor  S  should  be  studied  for  its  relationship  to 
complexity  measures. 

K.  This  study  provides  a  separate  growth  model  for  software  that 
can  be  combined  with  an  existing  hardware  reliability  growth 
model  for  a  given  time  r  of  interest.  The  development  of  a 
single  growth  model — perhaps  a  mathematical  merging  of  the  AMSAA 
model  and  the  Basic  Execution  Time  Model— -should  be  developed,  so 
that  a  single  growth  curve  can  be  used  for  the  combined 
hardware/software  system. 

L.  Section  2  of  this  report  provides  an  overview  of  the 
significant  issues  raised  in  modeling  MW  and  SW  systems  along 
with  limited  guidance  in  the  modeling  of  these  systems.  Research 
into  advanced  modeling  techniques  to  support  HW/SW  system  design 
is  needed.  This  research  is  needed  to  update  previous  RADC 
research  in  this  area  [James  et  al.  (1982)1. 

M.  Methods  of  instrumenting  software  code  to  ensure  that  any 
fault  which  may  occur  can  be  rapidly  identified  and  accurately 
isolated  to  the  section  of  the  code  (ie.  module)  which  caused  the 
problem  should  be  investigated.  Development  of  monitoring  methods 
which  can  ensure  improved  software  fault  isolation  with  minimum 
or  no  operational  penalty  will  help  accelerate  software 
reliability  growth  and  minimize  the  potential  for  maintenance 
induced  software  faults. 
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APPENDIX  A 
RELIABILITY  DATABASE 
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A. 


The  development  and  validation  of  the  techniques  presented  in 
this  report  required  the  collection  of  empirical  data  which  was 
organized  into  a  database.  The  database  contains  metrics  on  the 
characteristics  of  each  project's  software  development  process 
and  the  intermediate  work  products  that  emerged  from  the 
development  process.  Also  data  was  collected  on  the  software 
failures  that  occurred  during  each  project's  system  test  period. 
The  primary  use  of  the  data  was  to  develop  a  regression  model  for 
the  early  prediction  of  software  reliability. 

For  developing  the  predictive  equations  of  SOW  para.  4. 1.3.2, 
the  study  team  sought  data  from  a  set  of  projects  that  were 
diverse  in  size,  application,  and  development  strategy,  yet  had 
produced  similar  kinds  of  well-kept  documentation.  These 
projects  had  to  be  old  enough  to  have  started  system  test  (so 
that  failure  data  was  available) ,  yet  young  enough  that  project 
documentation,  management,  and  development  staff  were  still 
accessible  as  sources  of  information.  Six  projects  were  chosen 
from  which  to  collect  the  data. 

Many  projects  at  Hughes  already  collect  metrics,  similar  to  the 
24  metrics  used  in  this  study,  for  the  purposes  of  management 
reporting  and,  increasingly,  for  use  in  improving  the  software 
development  process.  The  Software  Engineering  Institute  (SEI)  at 
Carnegie  Mellon  University  is  under  contract  to  the  Department  of 
Defense  to  examine  the  quality  of  the  software  development 
process  used  by  potential  software  development  contractors.  SEI 
conducts  assessments  of  organizations'  software  engineering 
practices.  Organizations  are  graded  by  the  maturity  level  (1-5) 
that  SEI  determines  that  they  have  reached.  To  reach  level  4, 
the  organization  must  establish  and  maintain  a  set  of 
product/process  metrics  and  establish,  manage,  and  maintain  a 
database  of  those  metrics.  It  must  use  the  metrics  to  assess 
product  quality,  track  progress  toward  meeting  quantitative 
quality  targets,  and  compare  the  metrics  with  historical 
experience  on  similar  projects.  At  level  5,  the  metrics  are 
automatically  collected,  and,  in  the  spirit  of  Total  Quality 
Management  (TQM) ,  the  metrics  are  used  to  improve  the 
organization's  software  development  process  [Humphrey  (1990)]. 

Which  metrics  to  collect  for  the  purpose  of  software  reliability 
prediction  were  based  on  (1)  how  promising  the  metric  was  for 
predicting  software  reliability,  from  a  review  of  the  literature 
and  in  light  of  the  team's  collective  software  engineering 
experience;  and  (2)  the  availability  of  the  metric  across  the  six 
projects.  Furthermore,  to  be  useful,  each  metric  had  to  be 
easily  collectible  for  a  project  following  the  draft  MIL-HDBK 
and,  ideally,  represent  a  controllable  characteristic. 
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Most  influential  was  the  excellent  paper  by  Takahashi  and 
Kainayachi  (1989)  .  This  paper  presented  their  research  on  ten 
factors  that  could  influence  fault  density  and  concluded  that  the 
following  were  significant  factors:  frequency  of  changes  to 

program  specification,  programmers'  skill,  and  volume  of  program 
design  documents.  The  team  generalized  these  factors  to  three 
corresponding  claases  of  metrics:  (1)  volatility— how  often  work 
products  change  (whether  because  of  misunderstanding,  mistakes, 
or  fickleness)  ;  (2)  skill  and  effort  brought  to  bear  on  the 
development  task;  and  (3)  magnitude  of  the  task.  Twenty-four 
metrics  were  then  chosen  from  among  those  thr«*  categories. 
Ultimately,  the  inclusion  of  a  particular  ..  ric  was  a 
hypothesis.  The  metric  may  or  may  not  be  useful  for  predicting 
software  reliability.  Conversely,  there  are  undoubtedly  other 
metrics  that  wera  not  collected  that  could  have  been  useful  for 
prediction.  Indeed,  one  of  the  risks  of  the  study  was  that  the 
set  of  metrics  chosen  would  be  insufficient.  Fortunately  the  set 
of  metrics  did  turn  out  to  be  useful. 

The  Hughes  study  tea a  either  personally  performed  or  oversaw  the 
data  collection  process  on  each  project,'  so  that  consistent 
conventions  and  judgments  could  be  made  in  transforming  the  raw 
data  into  the  metrics.  To  encourage  cooperation  and  candor,  each 
project  was  promised  anonymity. 

The  six  Hughes  Ground  Systems  projects  selected  for  the 
collection  of  software  development  and  failure  data  are 
summarized  in  Table 
A-l. 

Table  A-l. 

SUMMARY  OF  PROJECTS  -  SOFTWARE  DEVELOPMENT  DATA 


PROJECT 

# 

TYPE 

# 

CSCIS 

KLOC 

LANGUAGE 

1 

Avionics 

1 

140 

JOVIAL 

2 

CSI 

3 

53 

Ada,  C, 
Assembly 

3 

Data  Comm 

3 

230 

Ada 

4 

Trainer 

1 

40 

Fortran 

5 

1 

7 

Fortran 

6 

Radar 

2 

295 

Fortran, 
ULTRA- 16 

For  the  Validation  task  two  additional  projects  were 
used  employed.  These  appear  in  Table  A-2. 
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Table  A-2.  Validation  Projects 


Essm 

Type 

#  CSCls 

KLOC 

E3ESHSH 

7 

Weapon 

1 

18 

Pascal 

8 

ISSffHI 

1 

120 

Ada 

Table  A-3  summarizes  the  metrics  that  were  collected  from  these  8 
projects.  The  first  column  shows  the  phase  the  data  originates 
in.  The  second  column  is  a  short  identification  code  for  the 
metric.  The  third  column  is  a  description  of  the  metric.  The 
fourth  column  is  the  class  of  metric  (l,  2,  or  3). 

The  study  team  was  successful  in  collecting  all  24  metrics  from 
all  CSCls  of  the  six  projects.  The  team  had  contingency  plans 
for  dealing  with  missing  data  but  did  not  need  to  invoke  them. 

The  failure  data  recorded  consisted  of  the  estimated  failure  rate 
at  the  start  of  system  testing,  adjusted  to  a  standardized 
average  execution  rate  of  three  million  instructions  per  second. 
Fault  data  was  in  the  form  of  program  trouble  reports  (PTRs)  and 
library  change  requests  (LCRs) .  Where  necessary,  calendar  time 
was  mapped  to  execution  time  and  randomization  [Musa  et  al. 
(1987)]  was  employed  to  minimize  the  effects  of  uncertainty  in 
failure  times. 

Metrics  were  normalized,  Where  applicable,  by  dividing  by  the 
number  of  kilo  lines  of  code. 
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TABLE  A- 3: 

PRODUCT/PROCESS  METRICS 


PHASE 


Requirement* 


Preliminary 

Design 


Detailed 

Design 


CODE  DESCRIPTION 


Errors  in  requirements  specif ioation  (SRS 


Requirements  statements  in  the  SRS 


Pages  in  the  SRS 


Man-months  for  requirements  analysis 


Requirements  ohanges  after  baseline 


Errors  in  preliminary  design  documents 


CLASS 


Computer  Software  Components  (CSCs) 


Units  (Ada  packages)  in  design  structure 


documents  (SDDs 


Man-months  for  pre limine 


Errors  in  design  documents 


Man-months  for  detailed  design 


Design  flaws  identified  after  baseline 


Design  flaws  after  an  internal  review 


Total  executable  lines  of  code  (log 


Faults  found  through  code  reviews 


Programmer  skill  level  (av 


Average  number  of  souroe  LOC  per  unit 


Average  number  of  branches  in  a  unit 


Percent  of  branches  covered 


Nesting  depth  average 


Times  a  unit  is  unit-tested 


Man-months  for  coding/unit  tact 


Defects  identified  through  walkthroughs  and 
reviews 


Historical  failure/fault  fisld  data  was  collacted  from  four 
projects,  summarized  in  Table  A-4. 


TABLE  A-4 : 
FIELDED  PROJECTS 


|  Project 

Type 

# 

CSCIS 

KLOC 

Language 

9 

Mobile 

Radar 

3 

150 

ULTRA-16 

10 

Air  Defense 
Radar 

3 

220 

ULTRA-16 

U 

Target 

Acquisition 

2 

280 

Fortran, 
ULTRA- 16 

12 

Air  Defense 
Radar 

3 

270 

ULTRA-16 

The  hazard  rate  per  fault  was  not  diaoernablo  from  the  available 
data.  The  PTRs  from  the  field  did  not  contain  information  about 
how  many  times  a  particular  failure  recurred.  Generally,  the 
user  community  eventually  devised  and  disseminated  among 
themselves  some  type  of  work-around  that  avoided  the  problem 
until  the  next  version  became  available.  The  study  team  pursued 
SOW  paragraph  4. 1.1.2  by  adopting  an  approach  that  allows  a 
program's  fault  hazard  rate  profile  to  be  determined 
experimentally.  The  fielded  systoms  were  not  close  enough 
geographically  to  be  used  in  the  validation  task  (section  7) ,  so 
ongoing  projects  were  employed  for  the  validation. 


A- 6 


APPENDIX  B 

CONSTANT  FAILURE  RATE  MODEL  FOR  SOFTWARE 
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B. 


Frozen  code  subjected  to  input  randomly  selected  from  a 
stationary  operational  profile  is  reasonably  modeled  as  a 
homogeneous  Poisson  process  (HPP) .  Such  a  stochastic  process  is 
characterized  by  a  constant  failure  rate  and  exponentially 
distributed  interfnilure  times.  This  appendix  provides  an 
axiomatic  derivation  of  the  constant  failure  late  model.  It  is 
shown  that  each  axiom  is  reasonably  satisfied  by  software. 

Let  I  be  the  input  apace  of  a  computer  program.  The  operational 
profile  assigns  to  each  possible  input  stats  iel  a  probability 
p(i)  of  bsing  sslsctsd,  with  £p(i)«l.  Ons  can  imagine  a  "test 
oracle'1  e(i)  associated  with  each  input  state  i  that  evaluates  to 
1  if  the  program  fails  when  executed  with  that  input  state,  and 
evaluates  to  0  if  the  program  succeads  [Kopetn  (1981), 
MacWilliamo  (1973)].  In  practical  situations,  the  values  of  the 
e(i)  's  will  not  be  known,  but  the  actual  values  are  immaterial  to 
this  discussion. 

t 

The  probability  of  failure  in  a  single  run  is  given  by 

0  •  J^p(i)  e(i)  (B-l) 

The  probability  of  success  is  R-l-Q.  The  reliability  of  the 
program  for  n  independent  runs  is  Rn.  The  probability  of  k 
failures  in  n  runs  is  governed  by  the  binomial  law 

B(n,k)  *(jj )paQ*m*  (B-2) 


This  model  is  called  a  "data  domain"  software  reliability  model 
because  it  employs  the  run  as  the  unit  of  exposure.  A  data 
domain  model  is  describsd  by  Nelson  (1973) .  Analogous  hardware 
reliability  models  use  kilometers,  cyolss,  and  missions  as  units 
of  exposure.  Most  hardware  and  software  reliability  models, 
however,  are  "time  domain"  models;  they  use  the  continuum  of 
time  as  the  failure-exposing  force. 

In  the  time  domain,  a  run  starting  from  input  state  i  has  a 
duration  t(i).  Like  the  e(i) 's,  the  t(i) 'a  would  not  in  gsnsral 
be  known  a  priori. 

To  transform  the  data-domain  model  into  the  time  domain,  five 
axioms  are  introduced.  It  will-  be  shown  that  the  data-domain 
model  satisfioe  the  axioms,  and  that  ths  conoegusnce  of  ths 
axioms  is  an  HPP. 
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Consider  the  counting  process  {N(t),t£0}.  N(t)  represents 
cumulative  number  of  software  failures  by  cumulative  execution 
time  t.  As  shall  be  proved  later,  if  the  following  five  axioms 
are  satisfied,  the  process  is  an  HPP  [Parzen  (1962)]: 

Axiom  1 — Initialization:  N(0)-0.  The  counting  of  failures 
begins  at  time  0. 

Axiom  2— Independent  increments:  N(t+a) -N(t)  is  independent 
of  N (t) . 

Axiom  3— o<Pr{N(t)  >0}<l,  t>0.  It  is  neither  impossible  nor 
certain  that  a  failure  will  occur  in  any  interval. 

Axiom  4 — 


Failures  do  not  occur  simultaneously 

Axiom  5— -Stationary  increments:  For  any  two  times  t  and  s 
and  any  At>o,  it  is  true  that 

Pr  (tf(  t+A  b)  -N(  t)  -m}  -  Pr  {s+ A  t )  -N(s)  «n}  (B-4) 


It  will  now  be  shown  that  the  data  domain  model  satisfies  the 
five  axioms. 

Axiom  1  merely  defines  when  the  counting  of  failures  begins. 

Axiom  2  says  that  the  number  of  failures  in  a  future  interval  is 
not  influenced  by  the  number  of  failures  in  the  past.  software 
satisfies  this  axiom  as  long  as  any  data  corruption  a  failure 
leaves  is  cleaned  up  before  resuming  execution. 

Axiom  3  says  that,  in  any  interval,  there  is  a  positive 
probability  that  a  failure  will  occur,  but  failure  in  the 
interval  is  not  a  certainty.  A  zero  probability  of  failure 
requires  perfect  software.  The  possibility  of  any  substantive 
piece  of  software  having  no  faults  is  exceedingly  remote. 

Axiom  4  says  that  only  one  failure  can  occur  at  a  time.  ANSI- 
IEEE  Standard  100-1988  defines  software  reliability  as  "the 
probability  that  software  will  not  cause  the  failure  of  a  system 
for  a  specified  time  under  specified  conditions."  This 
definition  implies  that  a  software  failure  is  a  crash,  hang,  or 
bad  output  that  causes  system  failure.  A  program  run  on  input 
set  i  will  either  cause  system  failure  [e(l)»l]  or  it  will  not 
[e(i)*0].  More  than  one  software  failure  cannot  occur  at  once, 
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because  a  software  failure  is  the  sum  total  of  what  the  program 
did  (or  did  not  do)  to  cause  one  system  failure. 


Axiom  5  says  that  the  distribution  of  failures  in  a  time  interval 
depends  on  this  length  of  that  interval,  not  on  when  the  interval 
begins.  For  software,  the  number  of  failures  in  an  interval 
depends  on  the  number  of  runs  that  occur  during  that  interval, 
not  on  when  tho  interval  begins.  Different  runs  might  take 
different  amounts  of  time,  but  the  probability  of  n  failures  in 
an  interval  depends  sole  on  the  probability  of  failure  per  run 
and  the  long-run  average  number  of  runs  that  occur  in  the  time 
interval . 

Every  non-negative,  integer-valued  random  variable  X  has  an 
associated  probability  generating  function  (pgf) .  There  Is  a 
ons-to-one  relationship  between  a  random  variable's  pgf  and 
probability  distribution.  Given  one,  the  other  is  uniquely 
determined  and  vice  versa.  The  pgf  is  a  power  series 
transformation  of  the  probability  mass  funotion  into  a  function 
of  the  (usually  complex)  variable  z.  It  is  defined  as 

Gx(z)  ■  5^  P* ,lX"x)z*  ■  E{z  *}  (B-5) 


The  dummy  variable  z  must  be  restricted  to  a  range  in  which  the 
power  series  is  convergent.  The  series  is  always  absolutely  and 
uniformly  convergent  for  |z|£l,  and  may  also  converge  for  a  more' 
extensive  range.  Hence,  G(z)  is  a  continuous  function.  The 
probability  mass  function  of  X  can  bo  recovered  by 


Pr  U-x} 


ajj?  (o) 

xl 


<»-«> 


which  is  the  x-th  derivative  of  Gx(z)  evaluated  at  z  evaluated  at 
z»0,  since  Gx(z)  is  the  Taylor  series  expansion  for  G„(z)  about 
z»0. 


If  X  and  Y  are  independent  random  variables,  the  pgf  of  their  sum 
is  the  product  of  their  pgf's: 

Gx.y(z)  ■  Gx(z)Gy(z)  (B-7 ) 


because 


Bt***}  -  E{z*}E{z  Y) 


(B-8) 


The  pgf  for  the  Poisson  distribution  is 
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<B-9> 


Gx(z)  ■  E[z  x)  -  £  exp  [ -A  t]  * x  * 

in  x 

=  exp[-At]£-<*^ 

*5o  x‘ 

Since  the  series 

Si*£*2*  (B-io) 

X! 

converges  to  Atz  for  all  AeO,  Equation  B-9  becomes 

exp [ -At]  exp  [A  tz]  •  exp  [At (z-l)  ]  (B-ll) 


To  prove  that  the  counting  process  {N(t),tkO}  satisfying  the 
axioms  is  a  HPP,  it  suffices  to  show  that  the  pgf  of  N(tj  is  that 
of  an  HPP  [Parzen  (1962)]: 

jff(z,  t)  ■  exp  [A  t  (r-1)  ]  ,  |zl<l  (B-12) 

The  proof  is  divided  into  two  parts.  The  first  part  shows  that 
the  pgf  is  a  consequence  of  assuming  that  a  positive  constant  A 
exists  satisfying  three  key  equations.  The  second  part  shows 
that  a  positive  constant  A  satisfying  those  key  equatir"  a  does  in 
fact  exist. 

The  three  key  equations  are 


lim  -• 

-Px{tf(At)-0}  _  t 

(B-13J 

Ac-0 

At 

lim  ■ 

Px  {iV(A  t)  *1)  .  ^ 

(B-14) 

AC-0 

At 

lim  ■ 

Pr  {//(At)  *2}  0 

(B-15) 

AC-0 

A  t 

From  the  Independent  increments  axiom  it  follows  that 

,  E{ztf<e.At)-m>}  EizNlt))  (B-l«) 
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From  ths  stationary  incrsmsnts  axiom  it  follows  that 

tfr(z,  fc+A t)  -  *(z,  C)  *U,At) 

and 

-rr  [*(*»  t+AC)  -*(z,  t)  1 
A  C 


It  is  now  shown  that 

lim-rr  {tjr(z,  A  t) -1}  »  X(z-l) 
Ae-o  A  C 


Expanding  *(z,At),  ons  may  writs 


£^*,At)  -1] 


.  [Pr{M(AO -01-1}  z-~rPr  (tf(At) -1} 
^  t  AC 


-  _ 


Pr  (tf(At)  -2} 


But,  for  | z | <1, 


zn  Pr  {tf(AO  -n)  r  Pr{W(At)^2} 


Thsrsfors,  to  obtain  Equation  B-l£ 
lim^  [*U,At)  -1] 

At-o  AC 

.  lim-^-[Pr{tf(AC) -0}-l3  +  limz-~rPr  {//(At) 
Ac»o  A  t  Ae-o  a  c 

+  llm~Pr{W(At)  i2)  «*  -k  +  zX  *  X(r-l) 

At-o  AC 


Nwxt  it  is  shown  that  from  Equations  B-18  and  B-19, 
is  obtained.  Lst  At-»0  in  Equation  B-18: 


(B-17) 


(B-19) 


(B-20) 


(B-21) 


-1}  (B-22) 


Equation  B-12 
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(B-23) 


lim-~  tt(z,  t+A  t) -  +  (z,  t)  ] 

AC-0  Ac 


■  limp(z,  C)  [ijr(z,  At)  -1] 
At-o  AC 


«  ijr{z,  t)  AU-l) 

The  pgf  satisfies  the  differential  equation  from  Equation  B-19 

-  AU-l)  f(z,  t)  ,  t*0  (B-24) 


with  initial  condition  t(z,0>*l.  The  solution  to  this 
differential  equation  is 

tfr(z,  t)  ■  exp[At(z-l)  ] ,  |z|<l  (8-23) 


Let  P,  (t)  denote  Pr{N(t)«j).  Assume  that  Pa(t) 
Since 

PrUWti+ta)  *0} 

«  Pr  {N(  t, + tj)  -N(  tx)  «0  A  Cx)  -0} 


is  bounded. 

(8-26) 


it  follows  from  the  axioms  of  stationary  and  independent 
increments  that 


P0(t^tt)  -  P0(t1)P0(ta> 


(B-27) 


If  for  some  arbitrary  time  t0,  P0(t0)-O,  and  P0(t)  is  bounded,  it 
can  be  shown  that 

P0(t)  -  exp  [-At]  (B-28) 

The  case  A-0  corresponds  to  a  degenerate  distribution  in  which  no 
events  ever  occur.  By  Axiom  3,  0<P0(t)<l  for  all  t,  so  Equation 
B-28  holds  with  A>0. 


a.  Next,  it  is  shown  that  from  Equation  B-28  one  obtains  the 
first  key  equation 


lim 

A  e-o  A  t 


■  lim 
A  e-o 


l-Pa(At) 

At 


.  Urn 

Ae-o  At 


(B-29) 


-  lim  Aexp[-AAt]  -  A 
Ae-o 
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Now  the  second  key  equation  (B-14)  can  he  proven  by  using  the 
fact  that 


l-P0(At) 
- It— 


L  .  0( At)  ' 
At  [  Px(Afc) 


(B-30) 


and 


litn-^^A- 

Ae-0  Pj  (At) 


(B-31) 


(by  Axiom  3),  where  Q(t)-Pr(N(t)22} .  From  Equations 
31,  and  the  first  key  equation  (B~13),  one  can  write 


A  ■  litn 

A  6-0 


l-P0(Afc) 

It 


lim 

At-0 


Pi  (At) 

IT~ 


i+ 


pTTAc) 


■  lim 
Ae-o 


Pi  (At) 

It~~ 


[1+0]  ■  lim 

A  6-0 


Pi  (At) 

ST“ 


B-30,  B 


(B-32) 


From  the  second  key  equation  (B-14)  and  from  Equation  B-31,  the 
third  key  equation  (B-15)  holds,  since 


-  limA^-lim-giM- 

A6-0  At  A  6-0  Pi  (At) 

■  lim  *  wm  Hi 

A  6-0  A  t 


-  lim^A^- 

Ae-o  At 


(B-33) 


Empirical  evidence  [Musa  (1979)]  also  supports  the  constant 
■  failure  rate  model  for  software. 

It  should  be  noted  that  the  way  failures  are  counted  for  the 
Basic  Execution  Tima  Model  during  growth  testing  is  difforent 
than  that  described  here.  Growth  testing  is  a  completely 
different  context.  Because  repair  activity  is  going  on,  the 
failure  rate  is  not  constant  but  varies  over  time.  The  Basic 
Execution  Time  Model  allows  for  multiple  simultaneous  "software 
failures,"  contrary  to  Axiom  #3.  During  growth  testing,  each 
discrepancy  (difference  between  the  output  variable  value  and  the 
value  dictated  by  the  requirements)  caused  by  a  distinct  fault  is 
counted  as  a  separate  software  failure.  The  reason  is  that  the 
debugging  will  address  all  of  the  discrepancies.  The  subsequent 
removal  of  each  underlying  fault  will  causa  a  decrement  in  the 
program  failure  rate. 

If  each  discrepancy  were  counted  as  a  separate  failure  during 
demonstration  testing,  the  result  would  not  be  an  HPP.  In  fact, 
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ths  distribution  of  failures  is  givsn  by  the  "stuttering  Poisson 
distribution"  [Haight  (1967)].  Axiom  4  doss  not  hold.  Lst  ths 
probability  of  k  software  failures  in  the  time  interval  (t,t+At) 

be 

A(3*dt  +  o(dt)  (B-34) 


Ths  probability  of  zero  failures  in  an  interval  is 

i6(t)  -  exp [-At]  (8-38) 

The  time  to  first  failure  is  thus  still  exponentially 
distributed.  The  probability  of  n+1  failures  in  time  t  is  given 
by  the  recurrence  relation  [Adelson  (1966)] 

(B-36) 


The  same  stuttering  Poisson  distribution  can  also  be  derived  by 
providing  a  finite  probability  ir  that  the  gap  between  successive 
failures  will  be  zero.  Then  the  cumulative  distribution  function 
of  the  gap  between  successive  failures  [Smith  (1957) ,  Smith 
(1958a),  Smith  (1958b)]  is 

Fix)  «  n  +  (l-n)  (l-exp[-Ax]) 


(8-37) 
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C.  DYNAMIC  ALLOCATION 


c.1.  Aiamagfeiang 


The  following  conditions  are  assumed  to  hold: 

i)  tha  componant  failuraa  ara  statistically  indapandant 

ii)  tha  failure  of  any  componant  rasults  in  failura  of  tha 
aggragata 

iii)  tha  componant  failura  rataa  ara  constant 


C.2.  Notation 

R*:  tha  raquirad  raliability  for  tha  aggragata  for  tha  mission 
tlma  t. 

1*:  tha  raquirad  failura  rata  for  tha  aggragata. 

R,(tf):  tha  allocatad  raliability  for  tha  i-th  componant 

n:  tha  total  numbar  of  componants 

m:  tha  total  numbar  of  tima  intarvala 

X,:  tha  allocatad  failura  rata  for  tha  i-th  componant 

t:  mission  tima  for  the  aggragata 

t'j:  tha  tima  at  which  tha  j-th  tima  interval  ends; 

ja0flf2f.«.m* 

t{ :  total  mission  tima  for  tha  i-th  componant;  i»l,2# . . . ,n. 


Note  tha  following  relationships: 

a)  t1+t2+. .  .+tn  -  t. 

b)  t0'  S  ti'  j  ...  S  t('. 

c>  V  -  t. 

d)  The  relationship  between  X*  and  F*  can  be  written  as 

R*  ( t)  - 

81  >.• .  -  WU*)  •  <0_1> 

c 

where  t  is  tha  aggragata  mission  tima. 
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c. 


3. 


nuuwahiuii  i. ammigi 


Suppose  that  an  acceptable  value  of  the  allocated  failure  rate 
for  each  component  i  must  belong  to  [A),00),  i-l,2,...n. 

By  using  the  proportional  allocation  technique  (see  Section  3  of 
this  report) ,  a  failure  rate  A,  can  initially  be  assigned  to  the 
i-th  component,  i-l,2,...n,  based  on  the  timing  relationship 
among  the  components  (sequential,  interleaved  or  simultaneous 
processes).  If  one  of  the  A, ’s  does  not  belong  to  its  accepted 
region,  without  loss  of  generality,  say  A,,  then  the  Effort 
Minimization  Algorithm  can  be  used  to  reallocate  the  failure  rate 
of  each  component  to  1,*,  i-l,2,...,n. 

Before  discussing  this  algorithm,  some  notation  needs  to  be 
introduced: 


A,. 

Ap 

Vs 


pre-allocatad  failure  rate  for  the  i-th  components; 

is™  1  tie 

the  smallest  acceptable  value  for  the  allocated  'failure 
rate  of  the  i-th  component;  i:»  l,...,n. 
new  allocated  failure  rate  for  the  i-th  component. 


0(1, , A, w):  effort  function;  the  amount  of  effort “needed  to 
decrease  the  failure  rate  of  the  i-th  component 
from  1,  to  1,  (i:*  1,  • . .  ,n)  • 


Note:  The  effort  function  G(x,y),  x  >  y  b  0,  is  assumed  to 

satisfy  the  following  conditions  [Kapur  et  al.  (1977)]: 

a)  G(x,y)  b  0. 

b)  G(x,y)  is  nonincreasing  in  y  for  a  fixed  value  of  x 
and  nondecreasing  in  x  for  a  fixed  value  of  y 

G(x,y)  b  G(x,y+ay) ,  ^  >  0. 

G(x,y)  S  G(x+ax,y)  <  >  0. 

c)  G(x,y)  is  additive;  i.  . ,  r  G(y,z)  -  G(x,z); 

z<y<x. 

d)  G(y,0)  has  a  derivative  h(y)  such  that  h(y)  is 
strictly  decreasing  in  y. 

e)  G(x,y)  -  G (tx, ty) ,  Vt  >  o. 

Four  different  effort  minimization  problems  will  be  presented  in 
light  of  the  timing  relationship  among  the  components 
(sequential,  interleaved  and/or  simultaneous  processes),  and  the 
complexity  factor  c,  of  each  component,  i-i,2,...n,  with 
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<C~3> 


SCj-1  ■ 


If  X.  <  Ai »  then  let  the  new  allocated  failure  rate  of  the  first 
component  be  JL.'  ■  Ai  >  £0,  at  least  one  of  the  remaining  X(, 

i-i,2,...n,  must  be  decreased.  To  do  this,  a  certain  amount  of 
effort  is  needed  (such  as:  further  engineering  development,  extra 
manpower,  extensive  testing,  etc.)  Let  the  effort  function  be 
G(X,,X,*),  i-l,2,...,n;  and  assume  this  function  satisfies  the 

above  conditions*  Now,  the  new  allocated  failure  rates  X(  ,, 
i-2,...,n,  can  be  calculated  by  solving  one  of  the  following 
minimization  problems. 

I)  Problem  I: 

If  the  aggregate  has  sequentially  active  components  and  the 
complexity  factors  are  not  taken  into  consideration  in  the 
objective  function,  then  the  new  allocated  failure  rates  for  the 
components  2  to  n  can  be  computed  by  solving  the  following 
optimization  problem: 


( p )  Minimize  VfiKX^XJ) 

such  t:hat  ljta+ . .  . . 

! 


(C-3) 


Rewriting  the  above  optimization  problem  (P)  yields: 

a 

(i»0  Minimize  ^TjGfX^Xj) 

& '  t .  ta  + . . .  +Xncn  ■  X 

X2  ”  Aa  3  0 

t 

«  "  A,  *  0 

-X2*  +  X2  z  0 

! 

rX; 


(C-4) 
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By  Kuhn-Tucker  conditions  [Phillips  at  al.  (1976)]  ,  there  exist 
, . . . , /U2n,2  And  1  such  that  the  following  relations  hold: 

(1)  lje2+.  . .  +Vnca-Xmt+Xlti  -  0 

(2)  la  -  A,  *  0 

I 

la  “  *  0 

-la  +  la  i  0 

! 

-la  +  la  *  0 


(3) 


l*i  (la  -V  "  0 

l  (C-») 

^a-l^a  “  ■  0 

Hfldla  +  la)  "  0 

l 

i*aa-a('la  +  la)  ■  0 


(4)  1*1,  ...,»*aa-a  *  0 

(5)  0'(la,la*)+lta-|*i+Ha  -  0 

0/(Xi,la)+At»-|*i+l*Ji»i  ■  0 

i 

O^ln*  Ij,) +1  tn-|*0.x+|*aa.a  ■  0 


Now,  multiplying  eaoh  formula  of  Equation  05  with  the 
corresponding  lk*  yields: 

(6 )  1*G,/ dk»  1*)  +1*1  tjp*“l***jf_i ■^‘IjtJ* jf*n-a  "  ®  (06) 


By  Equations  03  and  04,  one  of  the  following  cases  holds: 

(i)  if  Mk-1  >  0  then  lk*  -  ^  and  Mk+n.a  -  0. 

(ii)  if  Mk«.n.2  >  0  then  lk*  ■  lk  and  nk.:  ■  0, 

(iii)  Both  -  Mk,n.z  -  0. 


05 


Thus,  for  the  values  of  k  such  that  Mim  *  ^k*n-2 
can  be  rewritten  as 


0 ,  Equation  C-<5 


XkG'(Xk,X*k)  +1*1 1*  -  0 


(C-7) 


or 


1 


G'{Xk,X'k) 


(C-») 


Next,  assume  that 


Gd*,!*)  ■  t*lj) ,  for  all  tk>0,  k*l ,  2 , 


(C-») 


then,  1  -  -G'  (lktk,lk  tk)/tij  for  all  values  of  k's  such  that 
Mk.i  ■  M i»n.a  *  °*  Since  Gf(y)  is  a  strictly  decreasing  function 
1*  ■  c  /  tk  is  true,  for  all  values  of  k's  such  that 
M ■  Mjt+n.j  11  0,and  some  constant  c. 


Now,  without  loss  of  generality,  suppose  l1t1  £  l2t,  £ 
Next,  assume  for  the  first  k,  variables,  and  the 
variables,  case  (i)  and  (ii)  hold,  respectively,  then 


.  .  £ 

last 


^ntp* 

n-k2 


and 


lj  ■  I  •  •  •  ,  1*»  ■  1^ 
^kt  ■  #••••#  Xfl  ■ 


(C-10) 


where  k,  <  k-.  Under  these  assumptions,  a  pair,  k,  and  kj,  needs 
to  be  found  that  satisfies  the  following  relations: 

a)  A_ta+. .  .  +A*  t*  +  (n-JCj,- (n~k2)  -1)  c+1*  tk  +  . . .  +lnfcn  ■  X*t-Ltx 

b)  CiO  '  '  (C-ll) 

C)  A  £  ■•£■  £  1„,  V  iCi+1  £  m  £  Jca— 1 


From  (a) ,  it  is  true  that 


c 


Infill  * ' 

kj-ki-l 


(C-12 ) 


Substituting  the  above  equation  into  (c)  yields 
for  all  k,+l  £  m  £  k2-l. 

Therefore,  k,  and  k,  must  be  found  such  that  k,  <  k,  and  (7)  holds 
for  all  m,  where  k.+l  £  m  £  k2  -1.  One  choice  for  k.  and  k,  is  as 
follows: 


C-6 


(0-14) 


1)  Lat  kj  be  tha  minimum  value  of  indax  l  such  that 

Ajtj  <  . .  “Xata 


2)  Lat  k,  ba  tha  maximum  valua  of  indax  j  such  that 


(0-15) 


Hanca,  a  solution  of  (P)  can  ba  writtan  as: 
If  kj-l  d  k1  than 


foz  2  i  1  i  kx 
for  ka  i  i  &  n 


(C-16) 


alsa 


X 


i 


t'i' 

X’t-ij  -X^  c^,- 

"  :  ^-(ki+l) 


1  £  \ 
•  •  Ant;i 


for  2  ii  <  kx 
foz  kz  <  1  i  n 

foz  kt  i  i  i  k2 

(0-17) 


2 )  Problam  II : 

This  is  similar  to  Problam  I,  but  this  tima  tha  complexity 
factors  of  aach  component  ara  incorporatad  into  the  optimization 


(P)  to  compute  the  failure  rataa  for  tha  components  2  to  n.  (P) 
can  ba  rawrittan  as  follows: 


(P> 

such  that 


cj  4 


Minimize  ^/ciG(ki,kl) 

.  .  +kata  ■  I’t-ijtj, 
2b |  i  ^3 

I 

0,  i i alf  2 1 1  i  •  A 


Using  Kuhn-Tuckar  condition*  aa  in  tha  abova  problem, 
for  Problem  II  is  obtained  that  can  ba  written  aa: 

Let  kj  ba  tha  minimum  value  of  index  t  such  that 

c,  min  ci 

2  iisn 


Let  k,  ba  tha  maximum  value  of  index  j  such  that 


If  kj-l  S  k,  than 


,  J  if  2 siska 
if  Jc2<isn 


c-8 


<C-M) 


a  solution 


(C-19) 


(C-20) 


(C-21) 


else 


(Aj  if  2il&k1 
|A.i  if  k2*i&n 

^  A  *  tj-Aj  t2  “"’“A^  tk)-Xkt  ] 

if  J^+lziraJcj-l 


(C-22 ) 


III)  Problem  Hit 

If  the  aggregate  ha®  concurrently  active  component*,  and 
complexity  factors  are  not  taken  into  consideration  in  the 
objective  function,  then  the  new  allocated  failure  rates  for  the 
components  2  to  n  can  be  computed  by  solving  the  following 
minimization  problem! 


(P) 


Minimize  ^0(li,Ai) 

such  that  AJ+. . ,  +An  -  A'-Aj^ 

Aj  £  Aj  s  Aa 

i 

Afl  *  Afl  i  An 


(0-23) 


This  problem  has  the  same  objective  function  as  that  of  problem 
I,  but  the  first  constraints  in  these  two  problems  are  not  the 
same.  However,  the  Kuhn-Tucker  conditions  [Phillips  et  al. 
(1976)  ]  can  still  be  applied  to  obtain  a  solution  that  can  be 
written  as: 

Let  kj  be  the  minimum  value  of  index  t  such  that 

A|  <  A*-A1“A|-A,*1-.  ,  . -An  (C-24) 


Let  k1  be  the  maximum  value  of  index  j  such  that 


If  kj-l  £  Jc,  than 


•  •  / 2  £  i  £  k2 

Xi  t*i»  *2  <i  *  » 


(C-26) 


•Isa 


li#  2  £  i  £  kx 

X1,  iCj  ^  i  £  A 


/Ca-JCj-l 


(C-27) 

Jc^+1  £  i  £  Jc2-l 


IV)  Problem  IV: 

This  problem  is  similar  to  problem  III,  except  the  complexity 
factors  of  each  component  are  incorporated  into  the  objective 
function.  Therefore,  in  this  case  the  optimization  problem  can 
be  written  as 


(P)  Minimize  ^ciG(Xi,AJ) 

such  that  Xa+.  . .  +A«  ■  1*-^ 

xB 

c1  i  o,  i-i, 2, . .  .n 


E«.  -1 


i-i 


(C-28) 


Applying  the  Kuhn-Tucker  conditions  to  this  problem,  a  solution 
can  be  expressed  as  follows: 

Let  k2  be  the  minimum  value  of  index  l  such  that 
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A,  v  A.*-A1“X( - 

c,  min  Cjj 

2i  i&n 


(C-29) 


Let  k,  be  the  maximum  value  of  index  j  such  that 

Ai  s  - - 


<C-30) 


If  kg-l  :»  k,  then 


’  if  2*iska 
Xi  if  k2<lm 


(C**31) 


else 


*;■ 


(L  t  if  2*i£i?1' 
if  k2iiin 


‘W* 


if  Jq+1  s,  m  &  k2- 1 
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APPENDIX  D 


EARLY  PREDICTION  OF  PROGRAM  SIZE 


D-l 


This  appendix  discusses  some  of  the  methods  for  predicting  the 
size  of  a  software  program  prior  to  coding. 

The  methods  outlined  below  in  Parts  D.l,  D.2,  and  D.3  are  based 
on  the  psychometric  techniques  pioneered  by  Thurstons,  further 
developed  and  applied  by  Souder  and  Saaty.  These  methods  are 
employed  to  assign  a  position  to  each  module  along  an  interval 
scale.  Such  an  interval  scale  can  then  be  used  to  compute  the 
estimated  lengths  of  the  modules;  the  sum  of  these  estimated 
lengths  is  the  predicted  size  of  the  software  program  in  hand. 

In  order  for  the  methods  to  predict  the  absolute  size  of  each 
module,  the  set  of  modules  needs  to  be  augmented  by  at  least  two 
"reference"  modules.  These  are  modules  that  the  experts  are 
familiar  with  and  whose  sizes  are  Known.  These  reference  modules 
would  ordinarily  come  from  other  projects  in  the  organization. 
One  reference  module  must  be  the  shortest  of  all  the  modules. 

d.i  svfigflgaivff  Ratings  Method 

The  Successive  Ratings  Method  was  first  introduced  by  Souder 
(1980).  The  modules  are  first  ordered  according  to  their 
conceptual  lengths  (based  on  the  experts'  judgments  of  how  long 
or.  short  the  modules  are  relative  to  each  other)  .  Then,  the 
modules  can  be  entered  into  the  successive  ratings  form  (Table  D- 
1)  ,  based  on  their  order,  from  the  longest  to  the  shortest. 

Table  D-l:  Successive  Ratings  Method  Example 


Modules 

caiman  l 

Compare 

to 

longest 

Soiumn  Z 
Compare 
to 

shortest 

Saimaa  3 
Compare  to 
longest  and 
shortest 

1 

100 

120 

laa 

3 

60 

70 

65 

4 

30 

20 

25 

2 

20 

is 

_ a _ _ 

Note:  In  the  above  table,  module  1  is  the  longest  and  module  2  is 
the  shortest. 

The  numbers  100  and  10,  underlined  in  the  table,  are  pre-recorded 
parts  of  the  standard  successive  ratings  form. 
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In  the  first  column  each  lower-ordered  module  is  then 
successively  compared  to  the  highest-ordered  (longest)  module  and 
a  number  corresponding  to  its  relative  value  is  assigned.  For 
example/  the  number  60  for  module  3  in  Table  D-l  indicates  that 
the  length  of  module  3  is  60%  of  the  length  of  module  1. 
Similarly/  module  4  is  30%  as  long  as  module  1,  and  module  2  is 
20%  as  long  as  module  1. 

When  column  1  is  filled  in,  the  comparison  process  is  reversed  to 
obtain  the  data  in  column  2.  For  example,  module  4  is  judged  to 
be  twice  as  long  as  module  2;  module  3  is  seven  times  as  long  as 
module  2;  and  module  1  is  twelve  times  as  long  as  module  2. 

Each  number  in  column  3  should  reflect  the  values  of  the  modules 
relative  to  the  longest  and  shortest  modules. 

The  sequence  of  numbers  in  column  3  can  be  thought  of  as  an 
interval  scale  for  the  lengths  of  the  modules. 


D.2  Pairwise  Comparisons 

The  Pairwise  Comparisons  Method  was  developed  by  Saaty  (1982)  for 
solving  decision  making  problems.  A  judgmental  matrix  of  the 
relative  lengths  of  the  modules  (in  a  software  program)  is 
formed,  using  the  experts'  evaluation.  Then,  by  applying  either 
the  Geometric  Mean  Approach  or  the  Power  Method,  an  interval 
scale  for  the  lengths  of  the  modules  can  be  calculated. 

In  the  Pairwise  Comparisons  Method,  the  user  must  assign  numbers 
to  the  first  row  of  a  judgmental  matrix  of  the  form 


M1 

m2 

m3 

M4 

M1 

1 

M2 

1 

m3 

1 

M4 

1 

based  on  the  following  definition. 
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If  module  A  Is  being  compared  to  module  B  then 


Number 


Definition 


1 

3 

5 

7 

9 

2, 4, 6, 8 
Reciprocals 


A  is  equal  to  B 

A  is  moderately  longer  than  B 

A  is  longer  than  B 

A  is  very  much  longer  than  B 

A  is  extremely  longer  than  B 

Intermediate  values  between  two  adjacent 

judgments 

If  module  i  has  one  of  the  preceding 
numbers  assigned  to  it  when  it  is  com¬ 
pared  with  module  j ,  then  module  j 
has  the  reciprocal  value  when  it  is 
compared  with  module  i 


For  instance,  if  M2  and  M4  are  moderately  longer  than  Mlf  and  M3 
is  extremely  longer  than  S^,  then  the  first  row  of  matrix  A  can 
be  written  as 


M1 

m2 

m3 

m4 

Mi 

1 

3 

9 

4 

m2 

1 

M3 

1 

m4 

1 

Now,  the  remaining  entries  in  the  upper  triangle  of  the  matrix  A 
can  be  computed  from  the  information  of  the  first  row.  For 
example,  from  the  first  row  we  have  M,  -  3MX,  M3  -  9M1; 

M4  -  4MX;  thus,  M3  -  (9/3 ) M2  -  3M2,  M4  -  (4/3)H2,  and  M4 
■(4/9)M3.  Therefore,  the  upper  triangle  of  matrix  A  can  be 
written  as 
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The  value  of  each  antry  in  tha  lowar  triangle  of  matrix  A  is  the 
raciprocal  of  tha  valua  of  tha  symxnatric  alamant  with  raspact  to 
that  antry  in  matrix  A.  So,  A  is 


Next,  wa  will  computa  an  interval  scale  for  tha  lengths  of  tha 
modules  based  on  tha  judgmental  matrix,  by  using  either  the 
Geometric  Mean  Approach  or  tha  Power  Method. 

i)  Geometric  Mean  Approach: 

For  each  row  i  of  matrix  A,  i  -  1,2 ,3, 4,  take  tha  product  of  the 
ratios  in  that  row  and  denote  it  by  TL.  Calculate  the 
corresponding  geometric  mean  P^,  where  P^  ■  i  ■  1,2, 3, 4. 
Let  P  -  J^P*.  WS  normalize  tha  P^  (i.e.,  transform  them  so  that 
their  resultant  sum  equals  unity)  by  forming  p^  -  (Pj/P)  .  Each 
Pi  is  a  corresponding  value  assigned  to  module  i,  i  '■  1,2, 3, 4. 
Rearranging  the  pj's  in  descending  order  gives  us  an  interval 
scale  for  tha  lengths  of  the  modules. 
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For  example, 


Mi 

m2 

m3 

m4 

Ik 

Pi-dl)1'4 

Pi-Pi/P 

Mi 

1 

3 

9 

4 

108 

3.2237 

0.59 

M2 

1/3 

1 

3 

4/3 

4/3 

1.0746 

0.19 

m3 

1/9 

1/3 

1 

4/9 

0.01646 

0.3582 

0.07 

M  ' 

1/4 

3/4 

9/4 

1 

0.42188 

0.8059 

0.15 

P-5.4624 

So,  an  Interval  scale  of  the  lengths  of  M2,  M3,  M4  is 


Interval  Scale 

M,  0.59 
M2  0.19 
M4  0.15 
M3  0.07 


One  deficiency  of  the  Geometric  Mean  Approach  to  obtain  an 
interval  scale  is  that  it  can  reverse  the  order  of  the  lengths  of 
the  modules.  To  detect  this  phenomenon,  assume  the  lengths  of 
two  of  the  modules  are  known.  If  the  interval  scale  value  of 
module  A  is  longer  than  the  interval  scale  value  of  B,  but  the 
length  of  module  A  is  shorter  than  the  length  of  module  B,  then 
the  Geometric  Mean  Approach  has  reversed  the  order  of  the  lengths 
of  the  modules.  In  that  case,  the  Power  Method  must  be  used 
instead. 
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ii)  Power  Method: 


The  matrix  A  is  multiplied  repeatedly  by  an  arbitrary  vector,  say 
[1,1, 1,1].  At  each  step,  the  resultant  vector  is  normalized  by 
making  its  largest  component  equal  to  unity.  This  recursive 
procedure  can  be  stopped  when  the  following  relation  holds: 

awttniM  <«£«<</*> i-(JW j)>  <  e. 


where  e  can  be  any  predetermined  small  number,  say  e  «  0.0000001, 
and  denote  the  i-th  component  of  the  unnormalized  fk. 


Below  is  the  illustration  of  the  Power  Method  when  it  is  applied 
to  matrix  A. 


1 

3 

9 

4 

1 

•  M 

17 

1 

1/3 

1 

3 

4/3 

1 

5.666667 

-  17 

0.33333 

1/9 

1/3 

1 

4/9 

1 

1.888889 

0.11111 

1/4 

3/4 

9/4 

1 

1 

4.25 

0.25 

L 

— - 

m  m 

*1 


1 

3 

9 

4 

1 

4 

1 

1/3 

1 

3 

4/3 

0.33333 

n 

1.3333 

-  4 

0.33333 

1/9 

1/3 

1 

4/9 

0.11111 

0.4444 

0.11111 

1/4 

3/4 

9/4 

1 

0.25 

1 

0.25 

M  • 

. 

h. 

L _ 

f2 
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1 

3 

9 

r 

1 

•  - 

4 

i  ' 

1/3 

1 

3 

4/3 

0.33333 

SB 

1.3333 

■  4 

0.33333 

1/9 

1/3 

1 

4/9 

0.11111 

0.4444 

0.11111 

1/4 

3/4 

9/4 

1 

0.25 

1 

0.25 

mmm 

M  ■ 

*3 

Co,  [l,  0.3333,  0.1111,  0.25]t  gives  an  interval  scale  for  the 
lengths  of  the  modules  M^,  M2 ,  M3,  M4  to  be 


Infctral  Seals 

Mi  i 

Mj  0.33333 

M4  0.25 

M3  0.11111 


Notice  that  for  the  above  interval  scale  and  the  one  in  part 
(IX)  (i)  ,  one  can  be  converted  to  the  other  by  means  of  a  linear 
transformation.  For  example,  if  we  multiply  0.59  with  each 
element  in  the  above  interval  scale,  we  will  obtain  the  interval 
scale  in  part  (II) (i)  . 


d.3  Ealjad-Campaglions  Mithofl 


The  idea  of  tho  Paired  Comparisons  Method  was  developed  by 
Thurstone  (1959) .  In  the  Paired  Comparisons  Method  the  experts 
are  prompted  with  the  names  of  two  modules,  selected  at  random, 
and  asked  to  decide  which  one  in  his  estimation  is  the  larger. 
Several  experts  make  their  judgments  on  many  modules. 

The  first  step  is  to  summarize  the  responses  in  a  matrix.  Below 
is  an  example  of  such  a  matrix. 
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i 


Module  (k) 

M 

A 

B 

c 

D 

E 

m 

A 

.50 

.79 

.16 

.43 

.67 

a 

u 

B 

.21 

.50 

.03 

.21 

.25 

H 

C 

.84 

.97 

.50 

.76 

.PI 

m 

D 

.52 

.79 

.24 

.50 

.68 

KH 

E 

.33 

.75 

.19 

.32 

.50 

The  value  of  each  entry  In  the  table  above  indicates  the 
proportion  of  experts  who  judged  nodule  (k)  to  be  larger  in  size 
than  each  of  the  other  nodules  (1).  Note  that  the  diagonal 
elements  contain  proportion  .50  and  that  for  each  pair  Pjj  it 
must  be  the  case  that  Pj_j  +  Pj^  «  1.00.  J 

The  next  step  is  to  create  a  new  matrix  by  converting  each 
proportion  to  a  z-3core.  The  normal  deviate  is  used,  with  all 
proportions  over  .50  yielding  positive  z-scores  and  those  under 
yielding  negative  z-scores: 


0.00 

0.80 

-.99 

-.05 

.44  ' 

-0.80 

0.00 

-1.88 

.81 

-.67 

.99 

1.88 

0.00 

.71 

.88 

.05 

.81 

-.71 

0.00 

.47 

,  -.44 

.67 

-.88 

-.47 

0.00, 

(e.g.,  in  the  first  table  of  this  section,  79%  of  the  experts 
believe  that  module  B  is  longer  than  module  A;  to  find  the  z- 
score  which  corresponds  to  0.79,  a  value  of  z  needs  to  be 
calculated  such  that  Pr(Z  s  z)  -  0.79,  where  z  e  N(o,l). 
Therefore,  z  (or  z-score)  -  0.80.) 

To  obtain  an  interval  scale,  the  sums  and  arithmetic  means  of 
each  column  are  calculated.  The  column  sums  are 


-.21  4.17  -4.46  -62  1.12 


and  the  averages  are 
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-.04  .83  -.89  -.12  .22 


From  any  of  the  above  methods,  an  interval  scale  can  be  obtained 
for  the  lengths  of  the  modules  in  a  software  program.  For 
example,  in  Part(I),  an  interval  scale  was  obtained  for  the  four 
modules  in  a  software  system: 


Modules 

1 

3 

4 
2 


Interval  Scale 
100 
65 
25 
10 


Now,  assume  the  lengths  of  modules  3  and  2  are  known,  say  1000 
and  500,  respectively.  Then,  the  lengths  of  the  modules  l  and  4 
can  be  calculated  as  follows: 


the  length  of  module  4  ■  (25-10) 

«  636 , 

the  length  of  module  1  •  500 ^ Qm£\-ic  °j(10Q-3-0) 

■  1318. 


Below  is  the  general  formula  to  compute  the  estimated  sizes  of 
the  modules  in  a  software  program: 

Assume  that  there  exist  n  modules,  and  that  an  interval  scale  for 
the  lengths  of  the  modules  has  bean  obtained  by  the  Successive 
Ratings  Method.  Also,  assume  that  the  lengths  of  modules  i,j,k,n 
(denoted  by  L^, Lj , L^, Ln) ,  i<j<k<n,  are  known. 
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(obtained  from  the  4th 
column  of  the  Successive 
Ratings  Form) 

LOC 

-1,1 

• 

i 

V 

V 

.  ^ 

> 

Lk 

r 

xn 

Ln 

Then,  the  length  of  the  1-th  module  le 


Ai  "  > 


(Il~Ii)t  VlsKj,  Ini 

(lrla) ,  VJe<Kii 


Note:  Any  one  of  the  three  methods  in  Parts  I,  n  and  III  can  be 
used  for  producing  an  interval  scale  for  the  lengths  of  the 
modules.  The  method  to  use  depends  on  the  level  of  familiarity 
of  the  software  engineering  staff  with  the  software  concerned. 
The  Successive  Ratings  Method  requires  the  most  understanding  of 
the  software  compared  to  the  other  two  methods,  while  the  Paired 
Comparisons  Method  requires  the  least. 
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E.  MULTIVARIATE  REGRESSION  ANALYSES 


This  appendix  presents  a  detailed  description  of  the  multivariate 
regression  models  which  were  formulated  as  a  part  of  developing 
prediction  models  for  software  failure  rate  (SOW  4. 1.3. 2).  The 
appendix  is  organized  into  four  sections: 

1.  Presentation  of  the  notation  that  is  used  in  the  regression 
analyses . 


2.  A  detailed  discussion  of  the  multivariate  regression  analyses. 


3.  The  correlation  matrix  for  standardized  software  failure  rate 
(Y)  and  software  characteristics  (x1»...rx2S) 


4.  Printed  computer  outputs  of  stepwise  selection  analyses, 

stepwise  regression  model,  principal  component  analyses,  and 
least  squares  method  (using  principal  components)  for  each 
software  development  phase. 


Summary  results  of  these  analyses  are  provided  within  the 
appropriate  sections  of  the  main  body  of  the  report. 

Table  E-l  contains  the  data  set  for  nine  observations  (CSCIs) 
which  are  collected  from  the  reliability  database  described  in 
Appendix  A.  This  data  will  be  used  in  regression  analyses  in 
this  section. 

Some  of  the  independent  variables  were  normalized  based  on  kilo 
lines  of  executable  code  (KLOC) .  The  normalized  independent 
variables  are  xv  . . . .  ,xH,x16,x,4,x25.  The  formula  to  normalize  x, 
where  i  6  (1,2, ... ,14,16,24,25}  is 

x,  ■  (x,  /  x15)  •  1000 

Recall  that  x15  is  the  number  of  LOC  in  each  observation. 

For  each  phase  of  the  software  development  process,  a  new 

regression  model  was  formed  which  involves  Y  (software  failure 
rate  as  the  dependent  variable)  and  x, ,  i*  l,2,...,k  where  k  £  n, 
and  contains  more  software  characteristics  as  they  become 

available.  Therefore,  several  regression  models  were  formulated 
for  each  software  development  phase  -(such  as  requirements 

analysis,  preliminary  design,  detailed  design,  coding  and  unit 
test)  . 

The  scatter  plots  Y  (software  failure  rate)  versus  x,  (software 

characteristic),  i  *  l,2,...,n,  based  on  the  collected  data  from 
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Appendix  A,  were  drawn.  These  scatter  diagrams  suggest  that  the 
regression  models  can  be  linear  in  terms  of  all  the  variables 
except  x6,  x,,,  x.3,  x14,  x16,  x2J.  However,  both  linear  and 

nonlinear  regression  models  involving  software  failure  rate 
(dependent  variables)  and  the  software  characteristics 
(independent  variables)  described  above  were  developed  for  each 
phase  of  software  development. 

The  general  linear  regression  model  is  of  the  form 
f  -  a0  +  a1x1  +  a2x2  +  ...  +  a ltxk 

where  k  assumes  different  values  depending  on  the  software 
development  phase.  Similarly,  the  non-linear  regression  model 
for  each  phase  of  the  software  development  is  a  sum  of  the  linear 
combination  of  items  in  a  subset  of  <x1# . . .  ,x,,x7, . . .  ,x10,x12,x17, 

. . .  ,x22,x24,x25)  and  the  linear  combination  of  items  in  the  subset 
of  (l/x6,  1/x,,,  l/x13,  l/x14,  l/x16,  .l/Xjj};  the  sizes  of  these 
subsets  depend  on  the  phase  of  the  software  development.  Notice 
that  x.5  is  not  used  in  any  of  the  regression  models,  except  to 
normalize  appropriate  independent  variables,  since,  the  software 
failure  rate  i  should  not  depend  on  the  size  (x15)  of  the 
software. 
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Table  £-1.  Data  Set  Observations 
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60 _ 

1074 

10 


Table  E-l .  Data  Set  Observations  (con't) 
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In  addition  to  the  regression  models  that  are  developed  as  the 
software  development  advances,  other  regression  models  were 
formulated  by  examining  the  correlations  of  the  independent 
variables.  But,  only  linear  relationships  between  Y  and  x(,  i  = 
l,2,...,n,  are  examined  in  these  models. 


E.l  Multivariate  Linear  Regression  Model  Development 


The  correlation  matrix  of  x1,...,xll  and  y,  based  on  the  collected 
data,  can  be  found  on  pages  E-21  through  E-24.  Most  of  the 
variables  are  highly  correlated  to  one  another.  For  example, 
consider  the  sets  {  x,  ,  x2  ,  x3  ;  x#  ,  x9  ,  x23  ,  x  25  }  , 
{X4,Xj,  x7,  X10,X.2,X18,X^,  X,o>  X22r 

x24),  etc.  The  variables  in  each  set  have  high  correlation  to 
one  other. 

Several  prediction  models  will  now  be  systematically  presented. 
A  different  model  is  developed  for  each  phase  of  the  software 
development  life  cycle,  using  the  metrics  that  are  cumulatively 
available  at  that  stage.  Using  the  successive  models,  the  user 
can  update  the  prediction  at  each  design  phase  when  the  metrics 
from  that  stage  become  available.  After  system  test  begins, 
actual  failure  data  will  become  available,  and  the  results  of 
growth  testing  will  supersede  the  prediction  model. 

The  regression  is  performed  by  using  the  method  of  least  squares. 
Least  squares  finds  the  "best-fitting"  hyperplane,  where  best 
fitting  means  to  find  the  hyperplane  such  that  the  sum  of  the 
squares  of  the  deviations  of  the  predicted  Y  values  from  the 
observed  Y  values  is  a  minimum,  which  is  the  criterion  of  minimum 
variance  unbiased  estimation.  Because  the  number  of  metrics 
available  in  latter  phases  exceeds  the  number  of  observations, 
two  different  approaches  were  taken:  selecting  a  subset  of  the 
most  significant  predictors  using  stepwise  regression,  and 
aggregating  metrics  using  principal  components  analysis. 
Additionally,  a  regression  model  was  formulated  that  is  based  on 
the  correlations  of  the  independent  variables. 

Although  more  than  one  regression  model  was  developed,  only  one 
was  ultimately  chosen  by  comparisons  among  the  models  in  the 
validation  task.  That  model  is  the  one  that  will  be  used  In  the 
draft  MIL-HDBK. 

(Note  that  all  regression  analyses  are  done  based  on  a  0.15 
significance  level.) 
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Thar*  ara  only  f  iva  Indapandant  variables  involved  in  th* 
regression  nodal  for  the  requirements  analysis  phase. 


x.:  number  of  errors  in  the  requirements  specification 
(SRS) 

x2:  Requirements  statements  in  the  SRS 
x3:  Pages  in  SRS 

x4:  Man-months  for  requirements  analysis 
x5:  Requirements  changes  after  baseline 

Since  the  number  of  observations  is  greater  than  the  number  of 
independent  variables  in  this  case,  an  attempt  has  been  made  to 
produce  the  regression  model  by  using  the  method  of  least  squares 
(i.e.,  the  surface  of  the  regression  model  fits  the  corresponding 
set  of  data  points  such  that  the  sum  of  the  squared  errors  is 
smallest) .  The  result  of  this  technique  is: 

Y  -  5.126  +  15. 717x1  -  0.172x2  +  0.52xs  +  0.39x4  -  0.234X, 
with  Y  restricted  to  nonnegative  values. 


The  complete  output  of  the  above  regression  model  is  on  pages  E- 
25  and  E-26  of  this  appendix.  Based  on  this  output  one  can 
observe  that  the  value  of  the  coefficient  of  determination  (R2) 
is  rather  significant  («  0.85).  This  number  represents  the 
percentage  of  total  variation  in  the  failure  rate  sampled  that  is 
explained  by  the  regression  surface.  But,  since  the  p-values  of 
hypothesis  tests  (I)  and  (II)  are  all  higher  than  0.15  (the 
predetermined  significance  level) ,  and  the  values  of  the  standard 
deviations  of  the  parameter  estimates  are  also  quite  high,  it  is 
suggested  that  all  of  the  variables  x,,  x.,...,  x,  are  too 
insignificant  to  be  included  in  the  model.*  This  phenomenon 
occurred  as  the  result  of  too  few  data  points  (nino  observations) 
involved  in  the  regression  analysis  (with  five  independent 
variables) .  Therefore,  either  stepwise  regression  or  principal 
components  analysis  is  preferable. 


Most  of  the  regression  analyses  in  this  section  are  similar  to 
each  other.  The  list  of  formulas  for  the  items  that  appeared  in 
the  outputs  of  the  regression  analyses  were  taken  from  the 
SAS/STAT  user's  guide  and  are  included  in  this  appendix  starting 
on  page  E-18. 
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Using  the  stspwiss  regression  technique,  a  new  regression  model 
for  the  requirements  analysis  phase  involving  only  x,  and  x2  was 
developed.  It  can  be  written  as  follows: 

Y  -  11.46159363  +  12.5764363  X,  -  0.08453592  X2 

where  Y  is  restricted  to  nonnegative  values. 

(Note:  The  complete  output  of  the  stepwise  regression  analysis 
for  this  phase  is  provided  on  pages  £-27  through  E-30) 


The  value  of  Rz  of  each  variable  involved  in  the  final  stepwise 
regression  model  for  requirements  analysis  is  substantial,  and 
all  the  p-valuas  of  hypothesis  tests  (I)  and  (II)  are  smaller 
than  0.15.  Note  that  x.,  x2  and  x3  are  highly  correlated,  as  well 
as  x.  and  x.  (based  on  the  correlation  matrix  on  page  E-21) ,  and 
the  values  of  the  standard  deviations  of  the  parameter  estimates 
are  high  (for  y-intercept  and  xt) .  This  fact  suggested  that  the 
above  stepwise  regression  model  is  inferior  to  the  model  that  is 
produced  using  principal  components  analysis.  This  is  because 
stepwise  regression  does  not  work  well  with  correlated  variables, 
whereas  principal  components  does. 

Principal  components  are  linear  combinations  of  the  original 
variables?  i.e.,  if  P,  is  a  principal  component  of  (x1,...,x5), 
for  >1,2, . . .  ,5,  then  P,  is  of  the  form 

pi  ■  aii*i  +  aai*a  +  •  +  a5ixi 


The  coefficient  akJ  is  the  kth  component  of  the  ith  normalized 
eigenvector  of  the  sample  variance-covariance  matrix.  For 
example,  in  the  case  >1, 

P1  m  0  , Q0S548x1  +  0.99488x2  +  0.100704x3  -  0.000135x4  +  0. 000478*, 


The  list  of  eigenvalues  and  normalized  eigenvectors  of  the 
estimated  variance-covariance  matrix  for  the  requirements 
analysis  phase  is  provided  on  page  E-3i. 

Since  the  first  eigenvalue  is  greater  than  99.7%  of  the  sum  of 
all  eigenvalues,  it  is  only  needed  to  form  a  linear  regression 
model  involving  the  software  failure  rate  (dependent  variable) 
and  P1#  the  first  principal  component  (independent  variables). 

The  data  set  of  P1  was  formed  based  on  the  original  data  set  of 
x1(  x2,  ...,x5.  The  linear  regression  model  (based  on  principal 
components  analysis)  can  be  summarized  as 
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Y  -  18.04246146  +  0.05053379  P, 


where 

P1  ■  0.008548x1  +  0.99488*2  +  0.100704Xj  -  0.000135X4  +  0.000478x5 
and  Y  is  restricted  to  nonnegatlve  values. 

On  page  £-33/  the  output  of  the  least  squares  regression  analysis 
of  Y  and  P.  is  provided.  All  components  in  the  column  titled 
NProb  >  | T |  **  and  the  p-value  of  hypothesis  test  (I)  are  much  leee 
than  the  0.15  significance  level.  This  indicates  that  the  y- 
intercept  and  P1  variables  are  relevant  to  the  model.  The  minor 
disadvantages  of  this  model  are  that  R2  ■  0.4064,  (i.e.,  only 
40.7%  of  the  total  variation  in  the  failure  rate  sampled  is 
explained  by  the  regression  surface) ,  and  that  the  standard 
deviation  of  the  y-intercept  is  large. 

Although  the  stepwise  regression  method,  in  this  case,  gives  a 
model  with  better  values  for  R2  than  principal  components 
analysis,  the  fact  that  only  x1  and  x.  are  chosen  by  the  stepwise 
method  (even  though  these  two  variables  are  highly  correlated) 
suggests  that  the  stepwise  model  should  not  be  preferred. 


E. 1.1.2  Preliminary  Design  Phase 


In  addition  to  the  variables  identified  in  the  requirements 
analysis  phase,  five  more  variables  are  involved  in  the 
regression  analysis  in  this  phase.  They  are: 

xA:  number  of  errors  in  preliminary  design  documents 
xr:  number  of  computer  software  components  (CSCs) 
x8:  number  of  units  (or  Ada  packages)  in  the  design 
structure 

x9:  Pages  in  design  documents  (SDDs) 
x10:  Man-months  for  preliminary  design 

It  is  clear  that  a  valid  least  squares  regression  cannot  be 
obtained  here,  since  the  number  of  observations  is  less  than  the 
number  of  independent  variables.  So,  only  stepwise  regression 
and  principal  components  analysis  were  performed. 

The  stepwise  regression  technique  in  this  case  chooses  x1  and  x, 
to  be  in  the  final  stepwise  regression  model.  This  model  can  be 
written  as: 

Y  -  11.91767775  +  14.11517877  X,  -  0.13589515  X9 
where  Y  is  restricted  to  nonnegative  values. 
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The  output  for  stepwise  regression  analysis  for  the  preliminary 
design  phase  can  be  found  on  pages  E-35  through  E-38.  The  value 
for  R2  is  high,  and  the  components  in  the  column  titled  "Prob  > 
|  T  | " ,  as  well  as  the  value  of  "Prob  >  F",  on  page  E-37  are  all 
below  the  predetermined  level  of  significance.  Since  the 
stepwise  procedure  only  allows  a  variable  to  stay  or  enter  the 
regression  model  if  that  variable  meets  the  0.15  significance 
level,  the  fact  that  variables  Xj, . . .  ,x.,x10  are  not  in  the  final 
stepwise  model  shows  that  they  are  irrelevant  to  the  model. 

Since  there  are  10  independent  variables  involved  in  the  stepwise 
analysis,  and  only  9  observations  are  available,  the  MSE  for  the 
full  model  cannot  be  computed.  Therefore,  C  ,  the  Mallow's 
statistic  cannot  be  computed;  so,  only  R2  is  used  in  decision¬ 
making  for  the  model. 

From  the  correlation  matrix,  it  can  be  detected  that  most  of  the 
variables  in  {x1M..,x10}  are  highly  correlated  to  each  other, 
especially  x1  and  x$.  This  indicates  that  the  stepwise 
regression  model  may  not  be  good  in  terms  of  predicting  the 
software  failure  rate.  other  factors  used  to  examine  the 
validity  of  the  stepwise  regression  model  are  the  standard 
deviations  of  parameter  estimates.  Since  these  values  of  the  y- 
intercept  and  x.  are  high,  it  is  further  evidence  that  the 
stepwise  regression  model  may  not  be  adequate  to  describe  the 
relationship  between  software  failure  rate  and  software 
characteristics  which  are  available  after  the  preliminary  design 
phase. 

In  order  to  perform  principal  components  regression  analysis,  the 
eigenvalues  and  eigenvectors  of  the  estimated  variance-covariance 
matrix  of  x1 , . . . ,  x10  and  y  were  computed  and  are  shown  on  page  E- 
41.  Since  the  sum  of  the  first  two  eigenvalues  is  greater  than 
99.7%  of  the  sum  of  all  the  eigenvalues,  only  F1  and  P2  will.be 
involved  in  the  principal  components  regression  model.  Recall 
that 


i  0.998 


moans  99.8%  of  the  total  variation  of  the  sampled  software 
failure  rate  can  be  explained  by  only  two  principal  components, 
P1  and  P2. 

The  data  set  for  P1  and  P2  can  be  easily  computed  by  using  the 
following  formulas: 
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P.  »  0. 006892X1  +  0. 795885X,  +  0.07994X,  -  0.000288x4  +  0.000323xs 

+  0.000094X6  +  0.004297x7  +  0.012784X8  +  C.59995X,  +  O.OO2784x10 

P,  “  0.032916X,  +  0.448504X,  +  0.562345x3  +  0.115656x4  +  0.077519X, 

-  0.004974X6  +  0.071291X7  +  0.014141x8  -  0.671443X,  +  O.O77466x10 


The  output  of  the  least  squares  regression  model  for  Y,  P,  and  P2 
can  be  found  on  pages  E-42  and  E-43. 

The  value  of  R2  (the  coefficient  of  determination)  is  similar  to 
that  of  the  principal  components  regression  model  for  the 
requirements  analysis  phase.  For  a  0.15  level  of  significance, 
the  values  of  the  components  in  the  column  titled  "Prob  >  |  T  |  •* 
and  the  values  of  standard  deviations  of  parameter  estimates 
imply  that  only  the  y-intercept  and  P1  are  relevant  to  the 
regression  model.  Since  the  value  of  "Prob  >  F"  is  bigger  than 
0.15,  the  null  hypothesis  of  (I)  should  be  rejected  (i.e.,  all 
coefficients  of  independent  variables  must  be  0) . 

This  phenomenon  may  occur  as  the  result  of  achieving  too  large  a 
number  for  the  total  variation  of  software  failure  rate  which  is 
explained  by  the  regression  model.  Thus,  a  new  regression  model 
that  only  involves  Y  and  P1  was  considered  at  this  point.  The 
complete  output  of  this  regression  model  is  on  page  E>44.  Below 
is  the  summary  of  the  regression  model  with  Y  and  P.  as  dependent 
and  independent  variables,  respectively. 

Y  -  17.89874335  +  0.04016008  P, 


where 

P,  -  0 . 006892X,  +  0 . 795885X,  +  0.07994X,  -  0.000288X,  +  0.000323X, 
+  0.000094x6  +  0.004297x7  +  0.012784xg  +  0.59995X,  +  O.OO2784X10 

Y  is  restricted  to  nonnegative  values. 

This  model  possesses  very  good  statistical  characteristics.  For 
example,  all  p-values  of  hypothesis  tests  (I)  and  (II)  are  below 
0.15  and  the  value  of  the  coefficient  of  determination  R2  is  not 
too  small.  Also,  the  values  of  the  standard  deviations  of 
parameter  estimates  are  reasonable,  except  for  that  of  the  y- 
intercept . 
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Fcur  new  variables  are  introduced  for  this  phase: 

xn:  number  of  errors  in  design  documents 

x12:  Man-months  for  detailed  design 

x13:  Design  flaws  identified  after  baseline 

xu:  Design  flaws  identified  after  an  internal  review 


The  stepwise  regression  analysis  and  principal  components 
analysis  were  performed  to  obtain  different  regression  models. 
Then,  after  the  validation  task  is  completed,  one  of  these 
regression  models  will  be  chosen. 

From  stepwise  regression  analysis,  a  regression  model  can  be 
written  an  follows: 

Y  -  28.96925965  +  15 . 63285056X.  +  0.2540362X*  -  2.38561282X.  - 
0.08100319X9  -  ]  .1992758X11 

where  Y  is  restricted  to  nonnegative  values. 

The  output  of  the  stepwise  regression  analysis  for  the  detailed 
design  phase  can  be  found  on  pages  E-46  through  E-50.  Similar  to 
previous  phases,  the  valuo  of  Jr  is  high,  and  the  values  of  the 
components  in  the  column  titled  "Prob  >  |T|" ,  as  well  as  the  p- 
value  of  hypothesis  testing  (I)  are  relatively  small  compared  to 
0.15.  But,  since  high  correlations  exist  between  the  variables 
involved  in  the  stepwise  process,  and  the  valjes  of  the  standard 
deviations  of  parameter  estimates  are  high  (except  for  that  of 
x,)  the  stepwise  regression  model  is  not  good  for  predicting  the 
software  failure  rate,  after  the  detailed  design  phase. 

The  output  of  the  computed  eigenvalues  and  eigenvectors  of  the 
estimated  variance-covariance  matrix  for  x.,...,  xu,  Y  is 
provided  on  page  E-53.  The  first  three  principal  components  of 
{x^...,^}  are  selected  for  the  regression  model,  since  the  sum 
of  the  first  three  eigenvalues  is  greater  than  99.7%  of  the  sum 
of  all  the  eigenvalues  for  the  matrix. 

The  least  squares  regression  analysis  involving  Y,  P1,...,P3  can 
be  found  on  pages  E-55  and  E-56.  The  principal  components 
regression  model  for  the  detailed  design  phase  can  be  written  as 

Y  -  36.61996444  +  0.12930799  P1  -  2.4687751  P2  -  0.64069017  P3 

where 
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P.  -  0.006891X.  +  0.795821X,  +  0.07993X,  -  0.000289x4  +  0.000322x5 
+  0.000097X,  +  0.004296X,  +  0.012792XS  +  0.599912X,  +  0.002783x1Q  + 
0.10963X,,  +  0.001742X12  +  0.003708x,3  -  0.003289xu 


P,  «  0 . 01003  lX.  +  0.403242X,  +  0.485487X,  +  0.131773X,  +  0.07455X, 
+  0.018274X.  +  0 .  C77214x7  +  0.009831X,  -  0.595055X*  +  0.08573x.0  - 
0. 261383X11  +  0 . 113134x12  +  0.089211x13  -  0.348945xu 


P,  -  -0. 25458X-  +  0 . 016768x2  -  0.263973X,  +  0.202666X,  -  0.072133X. 
+  0. 62971X-  +  0.103537X,  -  0.120209X.  +  0.013861X,  +  0.106051x10  + 
0. 348472x11  +  0.111607^2  -  0.1D490lX13  -  0.490828xu 

where  Y  Is  restricted  to  nonnegative  values. 

Notice  that  in  this  case  the  value  of  R2  is  quite  high  and  the 
components  of  the  column  titled  "Prob  >  | T I "  (see  page  E-55)  are 
relatively  lover  than  0.15  (level  of  significance);  but  the 
values  of  tha  standard  deviations  of  y-intercapt,  P2  and  P3  are 
high.  Hence,  further  examination  is  required  to  validate  the 
credibility  of  the  above  model. 


E.i.i.4  Ceding,  anfl-Vnlt  ,1,0, at  -Phage 


Below  is  the  list  of  the  additional  variables  that  were 
considered  in  the  regression  analysis  after  the  Coding  and  Unit 
Testing  phase: 

x16:  Faults  found  through  coda  reviews 

x,7:  Programmer  skill  level  (average  years  of  experience) 

xjB:  number  of  units  undergoing  review 

x19:  Average  number  of  source  LOC  per  unit 

x20:  number  of  branches  in  a  typical  unit 

x21:  Percent  of  branches  covered 

x22:  Nesting  Depth  average 

x23:  number  of  times  a  unit  is  unit-tested 

x24:  Man-months  for  Coding  and  Unit  test 

x25:  Defects  identified  from  walkthroughs  and  reviews 

(Note:  x25  -  x13  +  x14  +  x16?  therefore,  only  x2$  was  considered  in 
the  regression  analysis.) 

As  in  the  previous  phases,  the  stepwise  regression  and  principal 
components  analyses  were  also  performed  for  the  coding  and  unit 
test  phase. 

From  stepwise  regression  analysis,  the  regression  model  can  be 
described  as  follows: 
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¥  «  23.91  +  16. 4x1  +  0.33x6  -  2.02xs  -  0.10X9  -  1.46X,,  -  1.33x17  - 
0.  r'04x18  +  12.99x21 

(In  case  the  calculated  value  of  Y  is  negative,  this  number 
should  not  be  used  as  the  predicted  software  failure  rate.) 


The  stepwise  regression  analysis  can  be  found  on  pages  3-57 
through  E-61.  Notice  that  the  P  values  for  the  selected 
variables  in  Step  8  are  extremely  large  (999999.99),  which 
indicates  that  this  model  is  not  reliable  in  describing  the 
relationship  between  software  failure  rate  and  software 
characteristics.  Despite  this,  an  attempt  was  made  to  perform 
the  least  squares  regression  analysis  based  on  Y,  x, ,  x.,  x.,  x«, 
x..,  x17,  x..,  x,..  The  results  can  be  found  on  page  E-62.  It  is 
not  surprising  that  this  attempt  failed  for  the  3ame  reason  as 
mentioned  above.  A  stepwise  regression  model  cannot  be  formed  at 
this  point,  and  the  model  was  therefore  discarded. 

Based  on  the  results  of  eigenvalues  of  the  estimated  variance- 
covariance  matrix  of  x^,  •  • .  ,x», ... ,  x^7, ... , x^ ,  only  the  iirst 
principal  component  is  selected  to  be  in  the  regression  model 
(for  the  principal  components  analysis  approach) . 

The  principal  components  regression  analysis  gives 

Y  -  38.02688694  -  2.58584501P, 

where 

P.  •  -0.00001X.  -  0.000851X,  -  0.00C137X,  -  0.000021x4  -  0.000015XS 
-  0.000025X.  -  0.000017X,  -  0.00002X.  -  0.000631Xp  -  O.OOO17x10  + 
0.000014X..  -  0.00002X.,  -  0.000012X17  +  0.005288X,. 

0.000125X19  -  0 .  OOOlO8X20  -  O.OOOXj,  +  0.000004x22  -  O.OOOOlSx^  - 
0.000019x24  -  0 .  OOOllTXjj 

and  Y  is  restricted  to  nonnegative  values. 

In  this  case,  the  value  of  R2  is  very  low  (0.0748,i.e. ,  only 
7.48%  of  the  total  variation  of  the  sampled  software  failure 
rates  can  be  explained  by  the  above  model)  .  Also,  since  the 
second  component  in  the  column  titled  "Prob  >  |T|"  is  much 

greater  than  0.15  (level  of  significance),  and  the  values  of  the 
standard  deviations  of  parameter  estimates  are  high,  V1  is  not 
"relevant'*  to  the  regression  model.  But,  P1  cannot  be  deleted 
from  the  model;  if  it  is  deleted,  there  will  be  no  model  to 
consider. 
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.1.2  Regression  Models  Based  on  Correlate 


Variables 


Since  the  regressors  x)(...xa  are  correlated,  it  is  difficult  to 
disentangle  the  effects  of  one  regressor  from  those  of  another, 
and  the  parameter  estimates  may  be  highly  dependent  on  which 
regressors  are  used  in  the  model.  Instead  of  formulating  new 
regression  model  -i  based  on  the  progress  made  in  software 
development,  another  way  to  approach  ths  task  of  finding  the 
relationship  between  software  failure  rate  and  software 
characteristics  was  used.  This  method  is  discussed  below. 

Independent  variables  are  assigned  to  different  sets  such  that 
the  elements  of  a  set  are  highly  correlated  to  one  another.  In 
this  case,  based  on  the  correlation  matrix  on  pages  E-21  through 
E-24,  it  was  determined  that  independent  variables  can  be  divided 
into  four  disjoint  sets  {  x^x^xj^x^x^x*  }  ,  (  x4,x5,x7,x10, 

/^19/Xjq,  Xjj , Xjj  }  ,  {  x^,Xj7  }  ,  {  x8  }  ,  {  Xj^  }.  (Some  of 
independent  variables  have  been  normalized,  so  that  the 
mere  effect  of  program  size  is  canceled  out. 

Next,  after  performing  principal  components  analysis  for  each  set 
with  more  than  one  element,  regression  models  can  be  formulated 
using  either  the  method  of  least  squares  or  stepwise  regression 
analysis.  It  was  determined  that  only  the  first  principal 
component  of  each  set  needed  to  be  involved  in  regression 
analyses,  since  the  fewest  number  of  independent  variables 
involved  in  regression  models  as  possible  is  desired.  (Note  that 
in  case  of  a  singleton,  the  element  in  that  set  is  its  first 
principal  component.) 

This  approach  was  expected  to  yield  reasonable  regression  models, 
since  the  independent  variables  involved  are  uncorrelated. 
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Notation: 


A 

B 

C 

D 

E 


{  X,,  Xj,  X3,  X8,  X9,  X23,  X25  ) 

{  x4,  Xj,  x7,  x10,  x12,  x18,  x19,  x28,  Xjj,  x24  } 

(  Xn,  x17  ) 

(  x6  } 

{  x21  } 


PA:  the  first  principal  component  cf  A 

PB:  the  first  principal  component  of  B 

Pc:  the  first  principal  component  of  C 

P„:  the  first  principal  component  of  D;  in  this  case  P0 

PE:  the  first  principal  component  of  E;  in  this  case  P£ 
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A  new  data  set  for  PA,  PB,  Pc,  PD,  PE  was  computed  based  on  the 
original  data  of  x1,...,xa. 


E. 1.2.1  Least  Squares  Methods  Applied  to  Principal  Components 


Below  is  the  formula  of  the  regression  model  involving  Y , 
PA,...,PE  which  is  produced  by  using  the  Least  Squares  method 

Y  -  303.72  +  0.049  PA  +  0.015  P,  +  0.041  Pc  -  0.647  P0  -  290.93  Pt 
where 

PA  -  0.006891X,  +  0.795466X,  +  0.079926x3  +  0.012782xa  +  0.599617X, 
+  0.008398X23  +  0.032006x25 

P.  »  -  0. 004142X-  -  0.002728X.  -  0.003202X,  -  0.003143X10  - 
0.003841x12  +  0.999722X.,  -  0.013048x.«  -  0.017678x,0  + 

0.00063X22  -  0.003 551x2a 

Pc  -  0.981742x11  -  0.190218X17 

PD  *  x6 
PE  -  X21 

and  Y  is  restricted  to  nonnegative  values. 

Although  the  value  of  R2  for  this  regression  model  is  large 
(0.7455) — i.e. ,  a  large  percentage  of  the  total  variation  in  the 
failure  rate  sampled  can  be  explained  by  the  regression  surface, 
the  p -value  of  the  hypothesis  test  (I)  and  most  of  those  of 
hypothesis  test  (II)  are  much  bigger  than  0.15  significance 
level.  This  suggests  that  all  the  coefficients  of  the 
independent  variables  in  the  regression  model  should  be  0.  Thus, 
the  model  is  not  adequate  to  describe  the  relationship  between 
software  failure  rate  and  its  characteristics. 

Notice  that  all  p-values  of  hypothesis  test  (II),  on  page  E-79, 
are  much  greater  than  0.15  (the  predetermined  significance 
level)  ,  except  for  the  p-value  of  the  hypothesis  test  for  PA. 
This  indicates  that  another  regression  model  which  only  involves 

Y  and  PA  needed  to  be  considered. 

This  model  can  be  summarized  as  follows: 

Y  -  17.88125956  +  0.04011814  PA 

where  PA  is  defined  previously.  (In  case  the  calculated  value  of 

Y  is  negative,  this  number  should  not  be  used  as  the  predicted 
software  failure  rate,  in  this  case.) 
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Although  the  value  of  Rz  is  smaller  than  that  of  the  regression 


model  which  involved  V,  PA, . . . ,  PE,  this  model  is  preferable 


since  the  all  p-values  of  both  hypothesis  tests  (I)  and  (II)  are 
lower  than  0.15  (the  predetermined  significance  level),  also,  the 
value  cf  the  standard  deviations  of  parameter  estimates  is 
reasonably  small. 


Since  PA  is  a  linear  combination  cf  ::2,  ::3,  xJ(  k,,  x25,  and 

uhe  complete  data  of  these  independent  variables  will  not  be 


available  until  after  the  Code  and  Unit  Test  phase  of  software 
development,  this  regression  model  cannot  be  used  in  the  early 
stages  of  software  development. 


E.l.2.2  Stepwise  Regression  Applied  to  Principal  Components 


Another  way  to  correctly  eliminate  some  of  the  independent 
variables  in  {P.,...,P8},  besides  examining  the  p-values  of 
hypothesis  test  (II),  is  to  perform  stepwise  regression  analysis. 

As  mentioned  earlier,  stepwise  regression  allows  a  variable  to 
enter  or  leave  the  regression  model  based  on  the  value  of  R2  and 
Mallow's  statistic  C  .  The  result  of  the  stepwise  regression 
analysis  in  this  casepis: 

V  -  27.78996571  +  0.03884431  PA  -  1.20635275  P„ 

where 

PA  -  0.006891X.  +  0 . 795466X,  +  0.079926X,  +  0.012782X.  + 
0.599617X,  +  0.008398X23  +  0.032006x25 

and  P0  -  x6.  (In  case  the  calculated  value  of  V  is  negative, 
this  number  should  not  be  used  as  the  predicted  software  failure 
rate,  in  this  case.) 

Although  the  values  of  R2  and  p-values  are  reasonably  good,  the 
values  of  the  standard  deviations  of  parameter  estimates  of  the 
y-intercept  and  P0  are  substantial.  Therefore,  this  regression 
model  is  not  adequate  in  term  of  describing  the  relationship 
between  software  failure  rate  and  software  characteristics. 


E.2  Multivariate  Nonlinear  Regression  Models  Development 


Based  on  the  scatter  plots  of  Y  versus  x{,  1  -  1,2,..., 25,  it  was 
noticed  that  Y  is  inversely  proportional  to  each  of  the  following 
independent  variables,  x6,  x1t,  x1?,  x.4,  xt.,  xri.  Therefore,  in 
this  section,  new  regression  models  of  software  failure  rate  and 
software  characteristics  are  formulated  based  on 


i  •  •  •  ,  X« ,  Xjf  «  .  .  ,  X^q/ X.« ,  Xu ,  X^,  •  •  •  ,  X»j,  Xg^,  Xjj  And  »/X/,  1/X<^  ,  1/X^jr 
l/x.4,  i/x16,  l/Xjj  •  (Recall  that  some  of  the  independent 

variables  were  normalized  based  on  KLOC.) 

There  were  more  than  one  regression  model  which  were  built  for 
each  phase  of  the  software  development  by  using  the  least  squares 
method,  stepwise  regression,  and  principal  components  analysis. 
The  number  of  regressors  involved  in  the  regression  analysis  for 
each  software  development  phase,  in  this  section,  is  the  same  as 
that  in  section  E.l. 

These  regression  models  are  almost  identical  to  or  possess  the 
same  characteristics  as  ths  corrssponding  modsls  in  the 
discussion  on  Multivariate  Linear  Regression  Modsls.  Therefors, 
creating  new  regression  models  based  on  the  inverse  proportional 
relationship  betweon  V  and  x,  ,  whsrs  i  belongs  to  (6,  11,  13 
,14,  16  ,23},  results  in  negligible  improvements  to  ths  linear 
regression  models.  Thus,  the  models  in  this  section  can  be 
ignored  in  the  process  of  validation. 
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£.3  NOTATION 

-  n:  number  of  observations 

~  number  of  independent  variables  in  a  regression  model 

-  y, :  actual  value  of  software  failure  rate  of  the  i-th 

observation;  ii-  1,2, . . . ,n. 

-  predicted  value  of  software  failure  rate  of  the  i-th 
observation;  i:-  l,2,...,n. 

“  b,  s  estimated  coefficient  of  the  i-th  independent  variable; 

i:*  0,1,2, ...,n;  b0:  estimated  y-intercept 
"  ®bi*  •■timated  standard  deviation  of  the  random  variable  b, ; 
!:■  0, 1,2, ... ,n  ' 


n 


Dep.  Mean  i  y  ■ 

n 


Sum  of  Squares  Model  •  SSR 


•  y  )2 


Sum  of  Squares  Error  -  SSE 

*  (  yt  ~  > a 


Sum  of  Squares  c  Total  •  SST 

■  SSR  +  SSE 


Mean  Square  Model  ■  MSTR 

a  SSR 
k 


Mean  Square  Error 


SSE 

n- (ic+1) 
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F  -  Value  (Model) 


MSTR 

MSE 


this  is  ths  F-statistic  of  ths  hypothssis  tasting  with 

(I)  H0i  b0  -  b,  -  . . .  -  bk  -  0 

H1 :  3  at  laast  ona  b,  >  b,  *  o  whara  0  s  i  s  k 

-  Prob  >  F  (Modal):  p-value  of  tha  abova  hypothasis  tasting  (I) ; 
this  is  tha  significance  probability,  or  tha  probability  of 
gatting  a  graatar  F-statistic  than  that  observad  if  tha 
hypothasis  in  (I)  is  true. 


Root  MSE  ■  sftfSE 


i 

Root  MSE  is  an  estimate  of  tha  standard  deviation  of  tha  error 
term. 

-  C.V:  coefficient  of  variation:  this  number  expresses  tha 
standard  deviation  of  ths  error  term  in  unitlass  value. 

C.V  -  (  100  *  Root  MSE  )  /  Dap.  Mean 

-  R2:  coefficient  of  determination;  a  measure  between  0  and  1 
which  indicates  tha  portion  of  the  (corrected)  total  variation 
that  is  attributed  to  the  fit  rather  than  left  to  residual  error. 

R2  -  SSR  /  SST 

-  Adj  R-sq:  the  adjusted  R2,  this  is  a  version  of  R2  that  has 
been  adjusted  for  degrees  of  freedom. 

Adj  R-sq  -  1  -  [((n  -  i)  (1  -  R2))  /  (n  -  p)  ]  ; 

where  i  is  equal  to  1  if  there  is  an  intercept,  0  otherwise;  and 
p  is  the  number  of  parameters  in  the  model. 


The  parameter  estimates  and  associated  statistics  are  all 
printed;  and  they  include  the  following: 

-  The  parameter  estimates 

-  the  Standard  Error,  the  estimate  of  the  standard  deviation  of 
the  parameter  estimate 
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-  T  for  H0  Parameter  -  0  gives  the  T-ratio  of  hypothesis  tests 
with 

(II)  H0:  b<  «  0 

H1 :  b,  *  0;  where  0  £  i  £  k 

T-ratio  ■  -£^- 

a»i 


-  Prob  >  J  T |  column  on  page  1 1 1 1 H  provides  the  p-value  for  each 
of  the  hypothesis  test  in  (II) .  This  is  the  two-tailed 
significance  probability,  or  the  probability  that  a  t  statistio 
would  obtain  a  greater  absolute  value  than  that  observed  given 
that  the  true  parameter  is  zero. 
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CORRELATION  MATRIX 


XI 

X2 

X3 

X4 

X5 

X6 

XI 

1.0000 

0.9362 

0.8452 

-0.3536 

-0.1201 

-0.2176 

X2 

0.9362 

1.0000 

0.9635 

-0.0651 

0.1014 

-0.0439 

X3 

0.8452 

0.9635 

1.0000 

0.1145 

0.2110 

0.0347 

X4 

-0.3536 

-0.0651 

0.1145 

1.0000 

0.6895 

0.4298 

X5 

-0.1201 

0.1014 

0.2110 

0.6895 

1.0000 

-0.1743 

X6 

-0.2176 

-0.0439 

0.0347 

0.4298 

-0.1743 

1.0000 

X7 

0.2094 

0.5114 

0.6461 

0.8147 

0.6155 

0.3814 

XB 

0.9073 

0.9670 

0.9496 

0.0130 

0.2251 

-0.0898 

X9 

0.9411 

0.9963 

0.9416 

-0.0932 

0.0891 

-0.0360 

X10 

-0.0041 

0.3049 

0.4660 

0.9189 

0.7586 

0.3108 

Xll 

0.4035 

0.3674 

0.2585 

-0.3284 

-0.3100 

0.4513 

X12 

-0.1623 

0.1224 

0.2788 

0.9134 

0.9108 

0.1270 

X1S 

-0.1006 

-0.3707 

-0.4807 

-0.6175 

-0.4669 

-0.3673 

X17 

-0.3104 

-0.1035 

0.0814 

0.7137 

0.4512 

-0.0085 

X18 

-0.0429 

-0.3204 

-0.4223 

-0.6001 

-0.4598 

-0.4065 

X19 

-0.4256 

-0.2358 

-0.0807  ' 

0.6879 

0.2103 

0.4592 

X20 

-0.3302 

-0.0665 

0.1197 

0.8457 

0.9220 

0.0152 

X21 

0.0368 

0.1637 

0.2210 

0.3051 

0.2667 

0.3339 

X22 

0.4056 

0.1948 

-0.0149 

-0.8757 

-0.7942 

-0.0875 

X23 

0.7783 

0.9375 

0.9648 

0.2588 

0.4084 

0.0107 

X24 

-0.2630 

0.0317 

0.2346 

0.9257 

0.8501 

0.2057 

X25 

0.5060 

0.6775 

0.7958 

0.5172 

0.2281 

0.4316 
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CORRELATION  MATRIX  (con’t) 


X7 

X8 

X9 

X10 

Xll 

X12 

XI 

0.2094 

0.9073 

0.9411 

-0.0041 

0.4035 

-0.1623 

X2 

0.5114 

0.9670 

0.9963 

0.3049 

0.3674 

0.1224 

X3 

0.S461 

0.9496 

0.9416 

0.4660 

0.2585 

0.2788 

X4 

0.8147 

0.0130 

-0.0932 

0.9189 

-0.3284 

0.9134 

X5 

0.6155 

0.2251 

0.0891 

0.7586 

-0.3100 

0.9108 

XS 

0.3814 

-0.0898 

-0.0360 

0.3108 

0.4513 

0.1270 

X7 

1.0000 

0.5406 

0.4829 

0.9626 

-0.0715 

0.8360 

X3 

0.5406 

1.0000 

0.9599 

0.3728 

0.3060 

0.2275 

X9 

0.4829 

0.9599 

1.0000 

0.2732 

0.4215 

0.0962 

X10 

0.9626 

0.3728 

0.2732 

1.0000 

-0.2305 

0.9476 

Xll 

-0.0715 

0.3060 

0.4215 

-0.2305 

1.0000 

-0.3468 

X12 

0.8360 

0.2275 

0.0962 

0.9476 

-0.3468 

1.0000 

X15 

-0.7474 

-0.3308 

-0.3567 

-0.7003 

0.0920 

-0.6153 

X17 

0.5725 

-0.0808 

-0,1668 

0.6689 

-0.8177 

0.6621 

X18 

-0.7104 

-0.2481 

-0.3140 

-0.6656 

0.0002 

-0.5899 

X19 

0.4956 

-0.1399 

-0;2796 

0.5772 

-0.3857 

0.5032 

X20 

0.6732 

0.0556 

-0.1020 

0.8372 

-0.4753 

0.9504 

X21 

0.3261 

0.0706 

0.1819 

0.2933 

0.3494 

0.2796 

X22 

-0.6003 

0.0643 

0.2346 

-0.7761 

0.5157 

-0.8821 

X23 

0.7499 

0.9465 

0.9234 

0.6047 

0.2108 

0.4580 

X24 

0.8059 

0.1392 

-0.0079 

0.9286 

-0.3908 

0.9692 

X2S 

0.8077 

0.6?00 

0.6493 

0.6798 

0.1599 

0.4740 
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CORRELATION  MATRIX 


X15  X17 


X18  X19  X20 


X21 


XI 

-0.1006 

-0.3104 

X2 

-0.3707 

-0.1035 

X3 

-0.4807 

0.0814 

X4 

-0.6175 

0.7137 

X5 

-0.4669 

0.4612 

X6 

-0.3673 

-0.0085 

X7 

-0.7474 

0.5725 

X8 

-0.3308 

-0.0808 

X9 

-0.3567 

-0.1668 

X10 

-0.7003 

0.6689 

Xll 

0.0920 

-0.8177 

X12 

-0.6153 

0.6621 

X15 

1.0000 

-0.5222 

X17 

-0.5222 

1.0000 

X18 

0.9756 

-0.4545 

X19 

-0.1376 

0.5969 

X20 

-0.5186 

0.6913 

X21 

-0.5707 

0.0000 

X22 

0.4671 

-0.7217 

X23 

-0.5400 

0.1381 

X24 

-0.6024 

0.7206 

X25 

-0.6567 

0.3210 

0.0429 

-0.4256 

-0.3302 

0.3204 

-0.2358 

-0.0665 

0.4223 

-0.0807 

0.1197 

0.6001 

0.6879 

0.8457 

0.4598 

0.2103 

0.9220 

0.4065 

0.4592 

0.0152 

0.7104 

0.4956 

0.6732 

0.2481 

-0.1399 

0.0556 

<0.3140 

-0.2796 

-0.1020 

<0.6656 

0.5772 

0.8372 

0.0002 

-0.3857 

-0.4753 

<0.5899 

0.5032 

0.9504 

0.9756 

-0.1376 

-0.5186 

>0.4545 

0.596* 

0.6913 

1.0000 

-0.0563 

-0.4951 

•0.0563 

1.0000 

0.4933 

•0.4951 

0.4933 

1.0000 

•0.7011 

-0.3709 

0.2078 

0.4395 

-0.5073 

-0.9286 

•0.4858 

-0.0231 

0.2782 

■0.5728 

0.6015 

0.9699 

•0.5954 

0.2408 

0.2867 

0.0368 

0.1637 

0.2210 

0.3051 

0.2667 

0.3339 

0.3261 

0.0706 

0.1819 

0.2933 

0.3494 

0.2796 

-0.5707 

0.0000 

-0.7011 

-0.3709 

0.2078 

1.0000 

-0.2887 

0.2343 

0.2492 

0.4182 
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CORRELATION  MATRIX  (con't) 


X22 

X2  3 

X24 

X25 

XI 

0.4056 

0.7783 

-0.2630 

0.5060 

X2 

0.1948 

0.9375 

0.0317 

0.6775 

X3 

-0.0149 

0.9648 

0.2346 

0.7958 

X4 

-0.8757 

0.2588 

0.9257 

0.5172 

xs 

-0.7942 

0.4084 

0.8501 

0.2281 

X6 

-0.0875 

0.0107 

0.2057 

0.4317 

X7 

-0.6003 

0.7499 

0.8059 

0.8077 

X8 

0.0643 

0.9465 

.0.1392 

0.6900 

X9 

0.2346 

0.9234 

-0.0079 

0.6493 

X10 

-0.7761 

0.6047 

0.9286 

0.6798 

Xll 

0.5157 

0.2108 

-0.3908 

0.1599 

X12 

-0.8821 

0.4580 

0.9692 

0.4740 

X15 

0.4671 

-0.5400 

-0.6024 

-0.6567 

X17 

-0.7217 

0.1381 

0.7206 

0.3210 

X18 

0  v  4395 

-0.4858 

-0.5728 

-0.5954 

X19 

-0.5073 

-0.0231 

0.6015 

0.2408 

X20 

-0.9286 

0.2782 

0.9699 

0.2867 

X21 

-0.2887 

0.2343 

0.2492 

0.4182 

X22 

1.0000 

-0.1353 

-0.9220 

-0.3584 

X23 

-0.1353 

1.0000 

0.3734 

0.7711 

X24 

-0.9220 

0.3734 

1.0000 

0.4698 

X25 

>0.3584 

0.7711 

0.4698 

1.0000 
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REQUIREMENTS  ANALYSIS  PHASE:  Laast  Squares  Mathod 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE  DF 

SQUARES 

SQUARE  f  VALUE 

PROB>F 

MODEL  5 

3099.68840  619 

.93768  3.195 

0.1839 

ERROR  3 

582.01382  194 

.00461 

C  TOTAL  8 

3681.70222  • 

ROOT  MSE 

13.92855  R- 

SQUARE  0.8419 

DEP  MEAN 

24.25556  ADJ  R-SQ  0.5784 

C.V. 

57.42418 

PARAMETER 

ESTIMATES 

PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1 

5.12618938 

13.38038672 

0.383 

XI 

1 

15.71786692 

8.32103039 

1.889 

X2 

1 

-0.17215913 

0.13550831 

-1.270 

X3 

1 

0.51992600 

0.85056011 

0.611 

X4 

1 

0.38956606 

2.53978622 

0.153 

X5 

1 

-0.23386176 

2 . 43926235 

-0.096 

VARIABLE 

DF 

PROB  >  { T | 

INTERCEP 

1 

0.7272 

XX 

1 

0.1553 

X2 

1 

0.2935 

X3 

1 

0.5842 

X4 

1 

0.8878 

X5 

1 

0.9297 
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OBS 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWER95% 

MEAN 

1 

1.53043 

36.0000 

29.8926 

6.8038 

8.2395 

2 

0.75 

31.0000 

15.3818 

8.9610 

-13.1366 

3 

9.82125 

64.0000 

64.0621 

13.9284 

19.7350 

4 

2.30714 

48.0000 

45.3538 

17.9698 

4.6773 

5 

1.1575 

10.0000 

23.1440 

6.8621 

1.3053 

6 

0.94162 

8.0000 

18.5746 

7.3890 

-4.9410 

7 

0 

6.5000 

3.8082 

13.8177 

-40.1666 

8 

0 

6.0000 

8.1040 

13.7796 

-35.7495 

9 

0 

8.8000 

9.3788 

13.8826 

-34.8026 

10 

13.6698 

• 

234.4 

111.5 

-120.5 

11 

1.47059 

• 

21.9711 

7.7646 

-2.7396 

UPPER95% 

LOWER95% 

UPPER95% 

OBS 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

1.53043 

51.5458 

-19.4409 

79.2262 

6.1074 

2 

0.75 

43.9001 

-37.3272 

68.0907 

15.6182 

3 

9.82125 

108.4 

1.3737 

126.8 

-0.0621 

4 

2.30714 

87.2304 

-14.6159 

106.5 

2.0462 

5 

1.1575 

44.9827 

-26.2713 

72.5592 

-13.1440 

6 

0.94162 

42.0902 

-31.6043 

68.7535 

-10.5746 

7 

0 

47.7830 

-58.6315 

66.2480 

2.6918 

8 

0 

51.9576 

-54.2503 

70.4584 

-2.1040 

9 

0 

53.5603 

-53.2066 

71.9643 

-0,5788 

10 

13.6698 

589.4 

-123.3 

592.1 

• 

11 

1.47059 

46.6817 

-28.7789 

72.7210 

• 

STD  ERR 

OBS 

ID 

RESIDUAL 

1 

1.53043 

12.1537 

2 

0.75 

10.6633 

3 

9.82125 

0.0653 

4 

2.30714 

5.0782 

5 

1.1575 

12.1209 

6 

0.94162 

11.8071 

7 

0 

1.7539 

8 

0 

2.0317 

9 

0 

1.1302 

10 

13.6698 

• 

11 

1.47059 

■ 

SUM  OF 

RESIDUALS 

1  W' 

^ 

7652E-14 

SUM  OF 

SQUARED  RESIDUALS  ! 

582.0138 

PREDICTED 

RES ID  SS  (PRESS) 

8041231 
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REQUIREMENTS  ANALYSIS  PHASE:  Stepwise  Selections 


STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 

WARNING:  2  OBSERVATIONS  DELETED  DUE  TO  MISSING  VALUES. 

NOTE:  SLENTRY  AND  SLSTAY  HAVE  BEEN  SET  TO 
.15  FOR  THE  STEPWISE  TECHNIQUE. 


STEP  1  VARIABLE  XI  ENTERED  R  SQUARE  -  0.67484762 

C(P)  -  1.17054544 


DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

1 

2484.58798182 

2484.587982 

14.53 

0.0066 

ERROR 

7 

1197.11424040 

171.016320 

TOTAL 

8 

3681.70222222 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

13.81140806 

XI 

5.69406301  1.49387362 

2484.587982 

14.53 

0.0066 

BOUNDS  ON 

CONDITION 

NUMBER: 

1, 

1 

STEP  2 

VARIABLE  X2  ENTERED 

R  SQUARE  ■ 
CfP)  - 

0.81369325 

0.53561689 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION  2  2995.77625834 
ERROR  6  685.92596389 
TOTAL  8  3681.70222222 

1497.888129 

114.320994 

13.10 

0.0065 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

XI 

X2 

11.46159363 

12.57643630  3.47633170 

-0.08453592  0.03997734 

1496.231406 

511.188277 

13.09 

4.47 

0.0111 

0.0789 

BOUNDS  ON  CONDITION  NUMBER:  8.100769,  32.40308 


NO  OTHER  VARIABLES  MET  THE  0.1500  SIGNIFICANCE  LEVLL  FOR  ENTRY 
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SUMMARY  OF  STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  V 


STEP 

VARIABLE 

ENTERED  REMOVED 

NUMBER 

IN 

PARTIAL 

R**2 

MODEL 

R**2 

C(P) 

1 

XI 

1 

0.6748 

0.6748 

1.17055 

2 

X2 

2 

0.1388 

0.8137 

0.53562 

STEP 

1 

2 

VARIABLE 

ENTERED  REMOVED 

XI 

X2 

F 

14.5284 

4.4715 

PROB>F 

0.0066 

0.0789 
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REQUIREMENTS  ANALYSIS  PHASE:  Stepwise  Regression  Analysis 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 
SUM  OF  MEAN 

SOURCE  DF  SQUARES  SQUARE  F  VALUE  PROB>F 

MODEL  2  2993.77626  1497.88813  13.102  0.0065 

ERROR  6  685.92596  114.32099 

C  TOTAL  8  3681.70222 

ROOT  MSE  10.6921  R-SQUARE  0.8137 

DEP  MEAN  24.25556  ADJ  R-SQ  0.7516 

C.V.  44.08103 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1 

11.46159363 

4.35386975 

2.633 

XI 

1 

12.57643630 

3.47633170 

3.618 

X2 

1 

-0.08453592 

0.03997734 

-2.115 

VARIABLE 

DF 

PROB  >  j  T | 

INTERCEP 

1 

0.0389 

XI 

1 

0.0111 

X2 

1 

0.0789 

OBS 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWER95% 

MEAN 

1 

1.53043 

36.0000 

30.3676 

5.1568 

17.7494 

2 

0.75 

31.0000 

19.3300 

3.8476 

9.9152 

3 

9.82125 

64.0000 

64.4400 

10.6835 

38.2983 

4 

2.30714 

48.0000 

40.0666 

7.1778 

22.5031 

5 

1.1575 

10.0000 

25.5450 

4.3932 

14.7952 

6 

0.94162 

8.0000 

22.1894 

3.9907 

12.4245 

7 

0 

6.5000 

6.3434 

5.4949 

-7.1021 

8 

0 

6.0000 

4.0762 

6.2383 

-11.1883 

9 

0 

8.8000 

5.9417 

5.6189 

-7.8072 

10 

13.6698 

• 

182.9 

45.6505 

71.1989 

11 

1.47059 

• 

24.9837 

3.8277 

15.6176 
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UPPER95% 

LOWER95% 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

1.53043 

42.9857 

1.3210 

59.4141 

5.6324 

2 

0.75 

28.7449 

-8.4751 

47.1351 

11.6700 

3 

9.82125 

90.5817 

27.4553 

101.4 

-0.4400 

4 

2.30714 

57.6301 

8.5553 

71.5779 

7.9334 

5 

1.1575 

36.2947 

-2.7400 

53.8300 

-15.5450 

6 

0.94162 

31.9543 

-5.7362 

50.1150 

-14.1894 

7 

0 

19.7889 

-23.0720 

35.7588 

0.1566 

8 

0 

19.3408 

-26.2139 

34.3663 

1.9238 

9 

0 

19.6906 

-23.6136 

35.4970 

2.8583 

10 

13.6698 

294.6 

68.1760 

297.6 

* 

11 

1.47059 

34.3497 

-2.8049 

52.7723 

• 

STD  ERR 

ID 

RESIDUAL 

1 

1.53043 

9.3664 

2 

0.75 

9.9758 

3 

9.82125 

0.4280 

4 

2.30714 

7.9246 

5 

1.1575 

9.7479 

6 

0.94162 

9.9194 

7 

0 

9.1721 

8 

0 

8.6836 

9 

0 

9.0967 

10 

13.6698 

• 

11 

1.47059 

• 

SUM  OF  RESIDUALS  1. 842976-14 
SUM  OF  SQUARED  RESIDUALS  685.926 
PREDICTED  RESID  SS  (PRESS)  76527.44 
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REQUIREMENTS  ANALYSIS  PHASE:  Principal  Component  Analysis 


PRINCIPAL  COMPONENT  ANALYSIS 

11  OBSERVATIONS 
5  VARIABLES 


SIMPLE  STATISTICS 


XI 

X2 

X3 

X4 

X5 

MEAN 

2.87712 

105.297 

20.0766 

2.83031 

2.59865 

ST  DEV 

4.52637 

243.706 

26.7754 

3.99664 

3.05383 

COVARIANCES 

XI 

X2 

X3 

X4 

X5 

XI 

20.488 

507.627 

76.7436 

-6.88532 

4.59364 

X2 

507.627 

59392.6 

6000.55 

-9.22677 

26.4418 

X3 

76.7436 

6000.55 

716.922 

11.1813 

23.2099 

X4 

-6.88532 

-9.22677 

11.1813 

15.9731 

5.96383 

X5 

4.59364 

26.4418 

23.2099 

5.96383 

9.32588 

TOTAL  VARIANCE-60155.27 


EIGENVALUE 

DIFFERENCE 

PROPORTION 

CUMULATIVE 

PRIN1 

60004.3 

59883.3 

0.997491 

0.99749 

PR1N2 

0.99950 

121.0 

98.2 

0.002012 

PRIN3 

0.99988 

22.9 

17.3 

0.000380 

PRIN4 

5.6 

4.2 

0.000093 

0.99998 

PRIN5 

1.00000 

1.4 

• 

0.000024 

E-32 


EIGENVECTORS 


PRIN1 

PRIN2 

PRIN3 

PRIN4 

PRIN5 

XI 

X2 

X3 

X4 

X5 

0.008548 

0.994880 

0.100704 

-.000135 

0.000478 

0.229267 

-.097623 

0.944222 

0.104725 

0.188107 

-.593818 

0.003010 

0.020841 

0.782781 

0.184906 

0.443844 

0.024948 

-.287915 

0.146934 

0.835401 

0.630671 

0.007280 

-.122369 

0.595565 

-.482213 
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REQUIREMENTS  ANALYSIS  PHASE:  Least  squares  Method 

(using  Principal  Components) 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 
SUM  OF  MEAN 

SOURCE  DF  SQUARES  SQUARE  F  VALUE  PROB>F 

MODEL  1  1496.13872  1496.13872  4.792  0.0648 

ERROR  7  2185.56350  312.22336 

C  TOTAL  8  3681.70222 

ROOT  MSE  17.66984  R-SQUARE  0.4064 

DEP  MEAN  24.25556  ADJ  R-SQ  0.3216 

C.V.  72.84864 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1 

18.04246146 

6.53814173 

2.760 

PI 

1 

0.05053379 

0.02308493 

2.189 

VARIABLE 

DF 

PROB  >  | T l 

INTERCEP 

1 

0.0281 

PI 

1 

0.0648 

OBS 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWER95I 

MEAN 

1 

4.31759 

36.0000 

18.2606 

6.4955 

2.9011 

2 

18.7646 

31.0000 

18.9907 

6.3621 

3.9467 

3 

839.563 

64.0000 

60.4687 

17.5602 

18.9451 

4 

5.83555 

48.0000 

18.3374 

6.4808 

3.0126 

5 

5.7755 

10.0000 

18.3343 

6.4814 

3.0082 

6 

13.2658 

8.0000 

18.7128 

6.4111 

3.5528 

7 

61.534 

6.5000 

21.1520 

6.0582 

6.8266 

8 

90.1651 

6.0000 

22.5988 

5.9384 

8.5567 

9 

67.323 

8.8000 

21.4445 

3.0283 

7.1898 

10 

9.03386 

• 

18.4990 

6.4503 

3.2462 

11 

59.2754 

• 

21.0379 

6.0706 

6.6831 
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OBS 


OBS 


UPPER95% 

LOWERS 5% 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

4.31759 

33.6202 

-26.2559 

62.7772 

17.7394 

2 

18.7646 

34.0347 

-25.4180 

63.3994 

12.0093 

3 

839.563 

102.0 

1.5618 

119.4 

3.5313 

4 

5.83555 

33.6621 

-26.1672 

62.8419 

29.6626 

5 

5.7755 

33.6605 

-26.1707 

62.8394 

-8.3343 

6 

13.2658 

33.8729 

-25.7353 

63.1610 

-10.7128 

7 

61.534 

35.4774 

-23.0184 

65.3224 

-14.6520 

8 

90.1651 

36.6410 

-21.4805 

66.6782 

-16.5988 

9 

67.323 

35.6993 

-22.7030 

65.5921 

-12.6445 

10 

9.03386 

33.7517 

-25.9809 

62.9788 

• 

11 

59.2754 

35.3927 

-23.1421 

65.2178 

• 

STD  ERR 

ID 

RESIDUAL 

1 

4.31759 

16.4326 

2 

18.7646 

16.4848 

3 

839.563 

1.9652 

4 

5.83555 

16.4384 

5 

5.7755 

16.4382 

6 

13.2658 

16.4657 

7 

61.534 

16.5988 

8 

90.1651 

16.6421 

9 

67.323 

16.6097 

10 

9.03386 

• 

11 

59.2754 

• 

SUM  OF  RESIDUALS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


3.46390E-14 

2185.563 

84362.07 
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PRELIMINARY  DESIGN  PHASE:  St«pwis«  Selection* 

STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 

WARNING:  2  OBSERVATIONS  DELETED  DUE  TO  MISSING  VALUES. 

NOTE:  SLENTRY  AND  SLSTAY  HAVE  BEEN  SET  TO 
.15  FOR  THE  STEPWISE  TECHNIQUE. 


STEP  1 

VARIABLE  XI 

ENTERED 

R  SQUARE  - 
C(P)  -  . 

0.67484762 

DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROR 

TOTAL 

1  2484 . 58798182 

7  1197.11424040 

8  3681.70222222 

2484.587982 

171.016320 

14.53 

0.0066 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

?ROB>F 

INTERCEPT 

XI 

13.81140806 

5.69406301 

1.49387362 

2484.587982 

14.53 

0.0066 

BOUNDS  ON 

CONDITION  NUMBER: 

1, 

1 

STEF  2 

VARIABLE  X9 

ENTERED 

R  SQUARE  - 
C  (P)  -  . 

0.86541068 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROF 

TOTAL 

2  3186.18444000 

6  495.51778222 

8  3681.70222222 

1593.092220 

82.586297 

19.29 

0.0024 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

FROB>F 

INTERCEPT 

XI 

X9 

11.91767775 

14.11517877 

-0.13589515 

3.07005978 

0.04662451 

1745.770281 

701.596458 

21.14 

8.50 

0.0037 

0.0268 

BOUNDS  ON 

CONDITION  NUMBER:  8. 

745719,  34. 

98288 

NO  OTHER  VARIABLES  MET  THE  0.1500  SIGNIFICANCE  LEVEL  FOR  ENTRY 
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SUMMARY  OF  STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 


VARIABLE  NUMBER 

PARTIAL 

MODEL 

STEP 

ENTERED 

REMOVED 

IN 

R**2 

R**2 

C(P) 

1 

XI 

1 

0.6748 

0.6748 

• 

2 

X9 

VARIABLE 

2 

0.190.6 

0.8654 

• 

STEP 

ENTERED  REMOVED 

F 

PROB>F 

.  1 

XI 

14.5284 

0.0066 

2 

X9 

8.4953 

0.0268 
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PRELIMINARY  DESIGN  PHASE:  Stepwise  Regression  Analysi 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DP 

SQUARES 

SQUARE 

F  VALUE 

PROB>F 

MODEL 

2 

3186.18444 

1593.09222 

19.290 

0.0024 

ERROR 

6 

495.51778 

82.58629704 

C  TOTAL 

8 

3681.70222 

ROOT 

MSE 

9.0877 

R-SQUARE 

0.8654 

DEP 

MEAN 

24.25556 

ADJ  R-SQ 

0.8205 

C.V. 

37.46647 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1 

11.91767775 

3.63650384 

3.277 

XI 

1 

14.11517877 

3.07005978 

4.598 

X9 

1 

-0.13589515 

0.04662451 

-2.915 

VARIABLE 

DF 

PROB  >  | T } 

INTERCEP 

1 

0.0169 

XI 

1 

0.0037 

X9 

1 

0.0268 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWER95% 

MEAN 

1 

1.53043 

36.0000 

33.3983 

4.8157 

21.6148 

2 

0.75 

31.0000 

19.3139 

3.2591 

11.3392 

3 

9.82125 

64.0000 

63.5265 

9.0808 

41.3065 

4 

2.30714 

48.0000 

43.9981 

6.6057 

27.8345 

5 

1.1575 

10.0000 

24.3466 

3.3913 

16.0483 

6 

0.94162 

8.0000 

18.6444 

3.1730 

10.8804 

7 

0 

6.5000 

3.2453 

5.0935 

-9.2181 

8 

0 

6.0000 

5.4633 

4.5832 

-5.7513 

9 

0 

8.8000 

6.3636 

4.3968 

-4.3949 

10 

13.6698 

• 

202.2 

39.9995 

104.4 

11 

1.47059 

• 

22.6830 

3.0574 

15.2017 
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UPPER95I 

LOWER95% 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

1.53043 

45.1818 

8.2323 

58.5643 

2.6017 

2 

0.75 

27.2887 

-4.3096 

42.9375 

11.6861 

3 

9.82125 

85.7466 

32.0908 

94.9623 

0.4735 

4 

2.30714 

60.1616 

16.5074 

71.4887 

4.0019 

5 

1.1575 

32.6449 

0.6119 

48.0813 

-14.3466 

6 

0.94162 

26.4084 

-4.9089 

42.1976 

-10.6444 

7 

0 

15.7086 

-22.2461 

28.7367 

3.2547 

8 

0 

16.6779 

-19.4414 

30.3680 

0.5367 

9 

0 

17.1221 

-18.3391 

31.0662 

2.4364. 

10 

13.6698 

300.1 

101.9 

302.6 

• 

11 

1.47059 

30.1643 

-0.7786 

46.1446 

• 

STD  ERR 

ID 

RESIDUAL 

1 

1.53043 

7.7069 

2 

0.75 

8.4832 

3 

9.82125 

0.3528 

4 

2.30714 

6.2411 

5 

1.1575 

8.4312 

6 

0.94162 

8.5158 

7 

0 

7.5261 

8 

0 

7.8473 

9 

0 

7.9533 

10 

13.6698 

i 

11 

1.47059 

• 

SUM  OF  RESIDUALS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


9 . 99201E-15 
495.517B 
99409.95 
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PRELIMINARY  DESIGN  PHASE:  Principal  component  analysis 


PRINCIPAL  COMPONENT  ANALYSIS 

11  OBSERVATIONS 
10  VARIABLES 


SIMPLE  STATISTICS 


XI 

X2 

X3 

X4 

MEAN 

ST  DEV 

2.87712 

4.52637 

105.297 

243.706 

20.0766 

26.7754 

2.83031 

3.99664 

X6 

X7 

X8 

X9 

MEAN 

ST  DEV 

7.11785 

8.39976 

2.11232 

2.56330 

8.02100 

4.62116 

90.035 

183.950 

COVARIANCES 

XI 

X2 

X3 

X4 

XI 

20.488 

507.627 

76.7436 

-6.88532 

X2 

507.627 

59392.6 

6000.55 

-9.22677 

X3 

76.7436 

6000.55 

716.922 

11.1813 

X4 

-6.88532 

-9,22677 

11.1813 

15.9731 

X5 

4.59364 

26.4418 

23.2099 

5.96383 

X6 

-13.2205 

5.33685 

-0.195098 

15.8104 

X7 

-0.495505 

327.701 

41.0176 

8.56419 

X8 

15.039 

949.037 

108.598 

-1.976 

X9 

391.763 

44639.8 

4421.27 

-34.1497 

X10 

-2.19708 

215.614 

30.1225 

9.80594 

X5 

2.59865 

3.05383 


X10 

2.05379 

2.65265 


X5 

4.59364 

26.4418 

23.2099 

5.96383 

9.32588 

6.66383 

3.46024 

3.52059 

12.0465 

4.3422 


E-40 


X6 

X7 

XI 

-13.2205 

-0.495505 

X2 

5.33685 

327.701 

X3 

-0.195098 

41.0176 

X4 

•15.8104 

8.56419 

X5 

-6.66383 

3.46024 

X6 

70.556 

9.12385 

X7 

9.12385 

6.5705 

X8 

-6.63936 

4.27394 

X9 

.  7.76958 

230.478 

X10 

8.10716 

6.54925 

X8 

X9 

X10 

15.039 

391.763 

-2.19708 

949.037 

44639.8 

215.614 

108.598 

4421.27 

30.1225 

-1.976 

-34.1497 

9.80594 

3.52059 

12.0465 

4.3422 

6.63936 

7.76958 

8.10716 

4.27394 

230.478 

6.54925 

21.3551 

721.786 

2.82043 

721.786 

33837.8 

144.485 

2.82043 

144.485 

7.03654 
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PRINCIPAL  COMPONENT  ANALYSIS 


TOTAL  VARIANCE«94098.56 


EIGENVALUE 

DIFFERENCE 

PROPORTION 

CUMULATIVE 

PRIN1 

93667.6 

93421.9 

0.995421 

0.99542 

PRIN2 

245.8 

157.5 

0.002612 

0.99803 

PRIN3 

88.2 

15.3 

0.-000938 

0.99897 

PRIN4 

72.9 

53.2 

0.000775 

0.99975 

PRIN5 

19.7 

17.1 

0.000209 

0.99995 

PRIN6 

2.6 

1.3 

0.000027 

0.99998 

PRIN7 

1.3 

1.0 

0.000014 

1.00000 

PRIN3 

•  0.3 

0.3 

0.000004 

1.00000 

PRXN9 

0.1 

C.l 

0.000001 

1.00000 

PRIN10 

0.0 

• 

0.000000 

1.00000 

EIGENVECTORS 


PRIN1 

PRIN2 

PRIN3 

PRIN4 

PRIN5 

XI 

0.006892 

0.032916 

0.366908 

0.150251 

-.254943 

X2 

0.795885 

0.448504 

-.211750 

-.308118 

-.079096 

X3 

0.079940 

0.562345 

0.444684 

0.560961 

-.068881 

X4 

-.000288 

0.115656 

-.208398 

0.157267 

0.588020 

X5 

0.000323 

0.077519 

0.152115 

0.114932 

0.472312 

X6 

0.000094 

-.004974 

-.671152 

0.629435 

-.310480 

X7 

0.004297 

0.071291 

-.112194 

0.094569 

0.296663 

X8 

0.012784 

0.014141 

0.185999 

0.094390 

0.017227 

X9 

0.599950 

-.671443 

0.214720 

0.329098 

0.112776 

X10 

0.002784 

0.077466 

-.111784 

0.086729 

0.397160 

PRIN6 

PRIN7 

PRIN8 

PRIN9 

PRIN10 

XI 

-.132385 

0.833557 

0.225988 

0.007562 

-.114912 

X2 

0.025944 

0.072973 

0.113615 

-.021007 

-.005793 

X3 

-.079061 

-.277114 

-.276344 

0.026034 

0.011202 

X4 

0.053866 

0.353501 

-.348731 

-.566503 

0.047860 

X5 

-.268552 

-.193451 

0.777313 

-.139691 

0.035931 

X6 

0.014661 

0.023119 

0.233663 

-.031726 

-.024293 

X7 

-.008912 

0.219785 

-.048807 

0.613538 

0.676878 

X8 

0.938085 

0.003176 

0.244888 

-.072321 

0.103053 

X9 

-.042748 

-.071209 

-.121775 

0.018832 

0.003802 

X10 

0.135658 

0.082206 

-.045238 

0.524752 

-.716705 
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PRELIMINARY  DESIGN  PHASE:  Least  Squares  Method 

(using  two  Principal  components) 


DEP  VARIABLE:  Y 


ANALYSIS  OF  VARIANCE 


SUM  OF 

SOURCE  DF  SQUARES 


MEAN 

SQUARE  F  VALUE  PROB>F 


MODEL  2  1478.65943. 
ERROR  6  2203.04281 
C  TOTAL  8  3681.70222 


739.32971 

367.17380 


2.014  0.2143 


ROOT  MSE 
DEP  MEAN 
C.V, 


19.16178 
24 . 25556 
78.99955 


R-SQUARE 
ADJ  R-SQ 


0.4016 

0.2022 


PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER- 0 

INTERCEP 

1 

17.90009142 

7.12992838 

2.511 

PI 

1 

0.04015744 

0.02001400 

2.006 

P2 

1 

0.01016278 

0.42464199 

0.024 

VARIABLE 

DF 

PROB  >  | Tj 

INTERCEP 

1 

0.0459 

PI 

1 

0.0916 

P2 

1 

0.9817 

OBS 


ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWER95% 

MEAN 

1 

4.07256 

36.0000 

18.0942 

7.2177 

0.4330 

2 

29.1337 

31.0000 

19.0161 

7.2326 

1.3186 

3 

1056.04 

64.0000 

60.2882 

19.0892 

13.5786 

4 

6.88223 

40.0000 

18.2343 

7.4892 

-0.0912 

5 

21.9383 

10.0000 

18.6229 

9.5496 

-4.7440 

6 

39.6677 

8.0000 

19.2342 

12.7410 

-11.9419 

7 

87.6421 

6.5000 

21.3631 

6.9391 

4.3838 

8 

100.738 

6.0000 

22.2260 

13.4384 

-10.6566 

9 

78.4684 

3.8000 

21.2210 

9.7140 

-2.5482 

10 

18.9421 

• 

18.7546 

8.0228 

-0.8766 

11 

91.6716 

• 

21.3924 

10.2067 

-3.5825 

E-43 


UPPER95I 

LOWER95% 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

4.07256 

35.7553 

-32.0090 

68.1973 

17.9058 

2 

29.1337 

36.7136 

-31.0998 

69.1321 

11.9839 

3 

1056.04 

107.0 

-5.8949 

126.5 

3.7118 

4 

6.88223 

36.5598 

-32.1069 

68.5755 

29.7657 

5 

21.9383 

41.9899 

-33.7643 

71.0102 

-8.6229 

6 

39.6C77 

50.4102 

-37.0718 

75.5401 

-11.2342 

7 

87.6421 

38.3424 

-28.5038 

71.2300 

-14.8631 

8 

100.738 

55.1086 

-35.0425 

79.4945 

-16.2260 

9 

78.4684' 

44 . 9*02 

-31.3469 

73.7889 

-12.4210 

10 

18.9421 

38.3857 

-32.0765 

69.5856 

• 

11 

91.6716 

46.3674 

-31.7316 

74.5164 

• 

STD  ERR 

ID 

RESIDUAL 

1 

4 • 07256 

17.7504 

2 

29.1337 

17.7444 

3 

1056.04 

1 • 66b? 

4 

6.88223 

17.6376 

5 

21.9383 

16.6126 

6 

39.6677 

14.3123 

7 

87.6421 

17.8612 

3 

100.738 

13.6595 

9 

78.4684 

16.5170 

10 

18.9421 

« 

11 

91.6716 

• 

SUM  OF  RESIDU/iLS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


9 . 50351E-14 
2203.043 
245131.3 


E-44 


PRELIMINARY  DESIGN  PHASE:  Least  Squares  Method 

(using  one  Principal  Component) 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 
SUM  OF  MEAN 

SOURCE  DF  SQUARES  SQUARE  F  VALUE  PROB>F 

MODEL  1  1478.44911  1478.44911  4.697  0.0669 

ERROR  7  2203.25312  314.75045 

C  TOTAL  8  3681.70222 

ROOT  MSE  17.74121  R-SQUARE  0.4016 

DEP  MEAN  24.25556  ADJ  R-SQ  0.3161 

C.V.  73.14286 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T 

FOR  ho: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1  17 

.89874335 

6.60114006 

2.711 

PI 

1  0 

.04016008 

0.01852996 

2.167 

VARIABLE 

DF  PROB  >  J  T  j 

INTERCEP 

1 

0.0301 

PI 

1 

0.0669 

PREDICT 

STD  ERR 

LOWER95t 

OBS 

ID 

ACTUAL 

VALUE 

PREDICT 

MEAN 

1 

4.07256 

36.0000 

18.0623 

6.5680 

2.5314 

2 

29.1337 

31.0000 

19.0688 

6.3796 

3.9832 

3 

1056.04 

64.0000 

60.3093 

17.6552 

18.5611 

4 

6.86223 

48.0000 

18.1751 

C .  5455 

2.6974 

5 

21.9383 

10.0000 

18.7798 

6.4308 

3.5732 

6 

39.6677 

8.0000 

19.4918 

6.3090 

4.5733 

7 

87.6421 

6.5000 

21.4185 

6.0569 

7.0961 

8 

100.738 

6.0000 

21.9444 

6.0091 

7.7350 

9 

78.4684 

8.8000 

21.0500 

6.0959 

6.6354 

10 

18.9421 

• 

18.6595 

6.4528 

3.4008 

11 

91.6716 

* 

21.5803 

6.0412 

7.2950 

E-45 


OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


ID 

4.07256 

29.1337 

1056.04 

6.88223 

21.9383 

39.6677 

87.6421 

100.738 

78.4684 

18.9421 

91.6716 


4.07256 

29.1337 

1056.04 

6.88223 

21.9383 

39.6677 

87.6421 

100.738 

78.4684 

18.9421 

91.6716 


UPPER95% 

MEAN 

33.5932 
34.1543 
102.1 
33.6528 
33.9864 
34- 1103 
35.7408 
36.1538 
35.4646 
33.9181 
35.8655 


16.4807 

16.5545 

1.7449 

16.4896 

16.5347 

16.5815 

16.6753 

16.6925 

16.6611 


LOWERS  5  % 
PREDICT 

-26.6719 

-25.5128 

1.1244 

-26.5406 

-25.8429 

-25.0335 

-22.9106 

-22.3483 

-23.3089 

-25.9810 

-22.7368 


UPPER95% 

PREDICT 

62.7965 

63.6503 

119.5 

62.8909 

63.4024 

64.0171 

65.7476 

66.2371 

65.4090 

63.2999 

65.8974 


RESIDUAL 

17.9377 

11.9312 

3.6907 

29.8249 

-8.7798 

-11.4918 

-14.9185 

-15.9444 

-12.2500 


STD  ERR 
ID  RESIDUAL 


SUM  OF  RESIDUALS  5.24025E-14 
SUM  OF  SQUARED  RESIDUALS  2203.253 
PREDICTED  RESID  SS  (PRESS)  148443.6 


E-46 


DETAILED  DESIGN  PHASE:  Stepwise  Selections 


STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 

WARNING:  2  OBSERVATIONS  DELETED  DUE  TO  KISSING  VALUES. 

NOTES  SLENTRY  AND  SLSTAY  HAVE  BEEN  SET  TO 
.15  FOR  THE  STEPWISE  TECHNIQUE. 

STEP  1  VARIABLE  XI  ENTERED  R  SQUARE  ■  0.67484762 

C(P)  -  . 

DF  SUM  OF  SQUARES  MEAN  SQUARE  F  PROB>F 

REGRESSION  1  2484.58798182  2484.587982  14.53  0.0066 

ERROR  7  1197.11424040  171.016320 

TOTAL  8  3681.70222222 

B  VALUE  STD  ERROR  TYPE  II  SS  F  PROB>F 

INTERCEPT  13.81140806 

XI  5.69406301  1.49387362  2484.587982  14.53  0.0066 

BOUNDS  ON  CONDITION  NUMBER:  1,  1 


STEP  2  VARIABLE  X9  ENTERED  R  SQUARE  •  0.86541068 

C(P)  -  . 

DF  SUM  OF  SQUARES  MEAN  SQUARE  F  PROB>F 

REGRESSION  2  3186.18444000  1593.092220  19.29  0.0024 

ERROR  6  495.51778222  82.586297 

TOTAL  8  3681.70222222 

B  VALUE  STD  ERROR  TYPE  II  SS  F  PROB>F 

INTERCEPT  11.91767775 

XI  14.11517877  3.07005978  1745.770281  21.14  0.0037 

X9  -0.13589515  0.04662451  701.596458  8.50  0.0268 

BOUNDS  ON  CONDITION  NUMBER:  8.745719,  34.98288 


E-47 


STEP  3 

VARIABLE  Xll  ENTERED 

R  SQUARE 
C(P)  -  . 

-  0.97400689 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROR 

TOTAL 

3  3586.00334120 

5  95.69888102 

8  3681.70222222 

1195.334447 

19.139776 

62.45 

0.0002 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

XI 

X9 

XII 

17.61432258 

14.26567506  1.47832193 
-0.12192015  0.02265278 
-0.88959153  0.19463790 

1782.310925 

554.427691 

399.818901 

93.12 

28.97 

20.89 

0.0002 

0.0030 

0.0060 

BOUNDS  ON 

CONDITION  NUMBER:  8.908018,  56 

.62403 

STEP  4 

VARIABLE  X8  ENTERED 

R  SQUARE 
C(P)  -  . 

-  0.99235831 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROR 

TOTAL 

4  3653.56779471 

4  28.13442751 

8  3681.70222222 

913.3919487 

7.0336069 

129.86 

0.0002 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

XI 

X8 

X9 

XII 

30.24108727 

14.41817104  0.89751810 
-2.32512285  0.75019785 
-0.06824032  0.02210312 
-1.04402729  0.12808083 

1815.147839 

67.564454 

67.042926 

467.340584 

258.07 

9.61 

9.53 

66.44 

0.0001 

0.0362 

0.0367 

0.0012 

BOUNDS  ON  CONDITION  NUMBER:  23.07832,  193.2544 


E-48 


STEP  5  VARIABLE  X6  ENTERED  R  SQUARE  -  0.99716673 

C(P)  -  . 


DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

5  3671. , 77097173 

734.2541943 

211.17 

0.0005 

ERROR 

3  3 

j . 43125049 

3.4770835 

TOTAL 

8  3681.70222222 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

28.96925965 

XI 

15.63285056 

0.82946548 

1235.079284 

355.21 

0.0003 

X6 

0.25403620 

0.11258423 

17.703177 

5.09 

0.1093 

X8 

-2.38561282 

0.52814664 

70.942420 

20.40 

0.0203 

X9 

-0.08100319 

0.01653809 

83.415842 

23.99 

0.0163 

Xll 

-1.19927580 

0.11332964 

389.373377 

111.98 

0.0018 

BOUNDS  ON  CONDITION  NUMBER:  ,  26.13549,  304.9335 


NO  OTHER  VARIABLES  MET  THE  0.1500  SIGNIFICANCE  LEVEL  FOR  ENTRY 
SUMMARY  OF  STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 


VARIABLE  NUMBER 

PARTIAL 

MODEL 

STEP 

ENTERED 

REMOVED 

IN 

R**2 

R**2 

1 

XI 

1 

0.6748 

0.6748 

2 

X9 

2 

0.1906 

0.8654 

3 

Xll 

3 

0.1086 

0.9740 

4 

X8 

* 

4 

0.0184 

0.9924 

5 

X6 

5 

0.0048 

0.9972 

VARIABLE 

STEP 

ENTERED  REMOVED 

F 

PROB>F 

1 

XI 

14.5284 

0.0066 

2 

X9 

8.4953 

0.0268 

3 

Xll 

20.8894 

0.0060 

4 

X8 

9.6059 

• 

0.0362 

5 

X6 

5.0914 

0.1093 

E-49 


DETAILED  DESIGN  PHASE:  Stepwise  Regression  Analysis 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DF 

SQUARES 

SQUARE 

F  VALUE 

PROB>F 

MODEL 

5 

3671.27097 

734.25419 

211.170 

0.0005 

ERROR 

3 

10.43125049 

3.47708350 

C  TOTAL 

8 

3681.70222 

ROOT 

MSE 

1.864694 

R-SQUARE 

0.9972 

DEP 

MEAN 

24.25556 

ADJ  R-SQ 

0.9924 

c.v. 

7.687698 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

Estimate 

ERROR 

PARAMETER* 0 

INTERCEP 

1 

28.96925965 

3.05969593 

9.468 

XI 

1 

15.63285056 

0.82946548 

18.847 

X6 

1 

0.25403620 

0.11258423 

2.256 

X8 

1 

-2.38561282 

0.52814664 

-4.517 

X9 

1 

-0.08100319 

0.01653809 

-4.898 

Xll 

1 

-1.19927580 

0.11332964 

-10.582 

VARIABLE 

DF 

PROB  >  J  T ! 

INTERCEP 

1 

0.0025 

XI 

1 

0.0003 

X6 

1 

0.1093 

X8 

1 

0.0203 

X9 

1 

0.0163 

Xll 

1 

0.0018 

E-50 


PREDICT 

STD  ERR 

LOWER95% 

ID 

ACTUAL 

VALUE 

PREDICT 

MEAN 

1 

1.53043 

36.0000 

37.4512 

1.2162 

33.5806 

2 

0.75 

31.0000 

30.5251 

1.8107 

24.7626 

3 

9.82125 

64.0000 

64.0698 

1.8639 

58.1379 

4 

2.30714 

48.0000 

46.7829 

1.4536 

42.1568 

5 

1.1575 

10.0000 

11.6845 

1.4654 

7.0209 

6 

0.94162 

8.0000 

6.2026 

1.2781 

2.1350 

7 

0 

6.5000 

6.4245 

1.2872 

2.3278 

8 

0 

6.0000 

5.6922 

1.3460 

1.4086 

9 

0 

8.8000 

9.4672 

1.8078 

3.7141 

10 

13.6698 

« 

212.8 

10.7106 

178.8 

11 

1.47059 

• 

15.1868 

2.2232 

8.1115 

UPPERS 5% 

LOWER95I 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

1.53043 

41.3218 

30.3661 

44.5363 

-1.4512 

2 

0.73 

36.2877 

22.2532 

38.7971 

0.4749 

3 

9.82125 

70.0018 

55.6791 

72.4606 

-0.0698 

4 

2.30714 

51.4089 

39.2584 

54.3073 

1.2171 

5 

1.1575 

16.3480 

4.1369 

19.2320 

-1.6845 

6 

0.94162 

10.2702 

-0.9920 

13.3972 

1.7974 

7 

0 

10.5211 

-0.7366 

13.6355 

0.0755 

8 

0 

9.9758 

-1.6267 

13.0111 

0.3078 

9 

0 

15.2204 

1.2019 

17.7326 

-0.6672 

10 

13.6698 

246.9 

178.2 

247.4 

• 

11 

1.47059 

22.2620 

5.9523 

24.4213 

i 

STD  ERR 

ID 

RESIDUAL 

1 

1.53043 

1.4135 

2 

0.75 

0.4454 

3 

9.82125 

0.0539 

4 

2.30714 

1.1680 

5 

1.1575 

1.1532 

6 

0.94162 

1.3578 

7 

0 

1.3491 

8 

0 

1.2905 

9 

0 

0.4573 

10 

13.6698 

• 

11 

1.47059 

• 

SUM  OF  RESIDUALS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


1 , 32561E-13 
10.43125 
7220.791 


DETAILED  DESIGN  PHASE:  Principal  component  analysis 


PRINCIPAL  COMPONENT  ANALYSIS 

11  OBSERVATIONS 
14  VARIABLES 


SIMPLE 

STATISTICS 

XI 

X2 

X3 

X4 

X5 

MEAN 

ST  DEV 

2.87712 

4.52637 

105.297 

243.706 

20.0766 

26.7754 

2.83031 

3.99664 

2.59865 

3.05383 

X6 

X7 

X8 

X9 

X10 

MEAN 

ST  DEV 

7.11785 

8.39976 

2.11232 

2.56330 

8.02100 

4.62116 

90.035 

183.950 

2.05379 

2.65265 

Xll 

X12 

X13 

X14 

MEAN 

ST  DEV 

7.29776 

8.21082 

2.43642 

3.64045 

2.65967 

4.22452 

8.51365 

9.09573 

COVARIANCES 


XI 

X2 

X3 

X4 

X5 

XI 

20.488 

507.627 

76.7436 

-6.88532 

4.59364 

X2 

507.627 

59392.6 

6000.55 

-9.22677 

26.4418 

X3 

76.7436 

6000.55 

716.922 

11.1813 

23.2099 

X4 

<6.88532 

-9.22677 

11.1813 

15.9731 

5.96383 

X5 

4.59364 

26.4416 

23.2099 

5.96383 

9.32588 

X6 

-13.2205 

5.33685 

-0.195098 

15.8104 

-6.66383 

X7 

-0.495505 

327.701 

41.0176 

8.56419 

3.46024 

X8 

15.039 

949.037 

108.598 

“1.976 

3.52059 

X9 

391.763 

44639.8 

4421.27 

-34.1497 

12.0465 

X10 

-2,19708 

215.614 

30.1225 

9.80594 

4.3422 

Xll 

0.731235 

785.26 

46.8588 

-6.66139 

-8.98976 

XI 2 

-3.84745 

139.957 

25.7475 

13.3912 

7.72505 

X13 

15.3442 

279.528 

62.0205 

-0.0315304 

3.65136 

X14 

-16.7971 

-206.739 

8.92737 

29.6529 

6.2227 

E-52 


PRINCIPAL  COMPONENT  ANALYSIS 


COVARIANCES 


X6 

X7 

X8 

X9 

X10 

XI 

-13.2205 

-0.495505 

15.039 

391.763 

-2.19708 

X2 

5.33685 

327.701 

949.037 

44639.8 

215.614 

X3 

-0.195098 

41.0176 

108.598 

4421.27 

30.1225 

X4 

15.8104 

8.56419 

-1.976 

-34.1497 

9.80594 

X5 

-6.66383 

3.46024 

3.52059 

12.0465 

4.3422 

X6 

70.556 

9.12385 

-6.63936 

7.76958 

8.10716 

X7 

9.12385 

6.5705 

4.27394 

230.478 

6.54925 

X8 

-6.63936 

4.27394 

21.3551 

721.786 

2.82043 

X9 

7.76958 

230.478 

721.786 

33837.8 

144.485 

X10 

8.10716 

6.54925 

2.82043 

144.485 

7.03654 

Xll 

34.3681 

0.495098 

6.72665 

662.567 

-2.72536 

X12 

5.65109 

7.941 

1.76349 

83.1111 

9.17345 

X13 

4.16506 

2.2108 

9.27088 

199.47 

0.768962 

X14 

34.3645 

13.8536 

-8.64942 

-239.719 

16.2101 

Xll 

X12 

X13 

X14 

XI 

0.731235 

-3.84745 

15.3442 

-16.7971 

X2 

785.26 

139.957 

279.528 

-206.739 

X3 

46.8588 

25.7475 

62.0205 

8.92737 

X4 

-6.66139 

13.3912 

-0.0315304 

29.6529 

X5 

-8.98976 

7.72505 

3.65136 

6.2227 

X6 

34.3681 

5.65109 

4.16506 

34.3645 

X7 

0.495098 

7.941 

2.2108 

13.8536 

X8 

6.72665 

1.76349 

9.27088 

-8.64942 

X9 

662.567 

83.1111 

199.47 

-239.719 

X10 

-2.72536 

9.17345 

0.768962 

16.2101 

Xll 

67.4176 

-7.23447 

-1.97936 

-13.1373 

X12 

-7.23447 

13.2529 

-0.967642 

21.8739 

XI 3 

-1.97936 

-0.967642 

17.8466 

0.119597 

X14 

-13.1373 

21.8739 

0.119597 

82.7322 

E-53 


PRINCIPAL  COMPONENT  ANALYSIS 


TOTAL  VARIANCE-94 279.81 


EIGENVALUE 

DIFFERENCE 

PROPORTION 

PRIN1 

93681.5 

93378.8 

0.993654 

PRIN2 

302.6 

170.3 

0.003210 

PRIN3 

132.4 

45.4 

0.001404 

PRIN4 

87.0 

42.8 

0.000923 

PRIN5 

44.2 

28.8 

0.000469 

PRIN6 

15.5 

3.5 

0.000164 

PRIN7 

12.0 

8.8 

0.000127 

PRIN8 

3.2 

2.0 

0.000034 

PRIN9 

1.2 

1.0 

0.000013 

PRXN10 

0.2 

0.2 

0.000002 

PRIN11 

0.0 

0.0 

0.000000 

PRIN12 

0.0 

0.0 

0.000000 

PRIN13 

0.0 

0.0 

0.000000 

PRIN14 

0.0 

• 

0.000000 

EIGENVECTORS 

PRIN1 

PRIN2 

PRIN3 

PRIN4 

XI 

0.006891 

0.010031 

-.254580 

0.269774 

X2 

0.795821 

0.403242 

0.016768 

-.311843 

X3 

0.079930 

0.485487 

-.263973 

0.601541 

X4 

-.000289 

0.131773 

0.202666 

0.030697 

X5 

0.000322 

0.074550 

-.072133 

0.123923 

X6 

0.000097 

0.018274 

0.629710 

0.333602 

X7 

0.004296 

0.077214 

0.103537 

0.026186 

X8 

0.012783 

0.009831 

-.120209 

0.132078 

X9 

0.599912 

-.595055 

0.013861 

0.314393 

X10 

0.002783 

0.085730 

0.106051 

0.012106 

Xll 

0.010963 

-.261383 

0.348472 

0.229051 

X12 

0.001742 

0.113134 

0.111607 

0.001101 

X13 

0.003708 

0.089211 

-.104901 

0.329080 

X14 

.  -.003289 

0.348945 

0.490828 

0.001316 

CUMULATIVE 

0.99365 

0.99686 

0.99827 

0.99919 

0.99966 

0.99982 

0.99995 

0.99999 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 


PRIN5 

-.048789 

-.275479 

-.036734 

0.299039 

0.284615 

-.290808 

0.138031 

0.100100 

0.376196 

0.199331 

-.409717 

0.355924 

-.124027 

0.380947 
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PRINCIPAL  COMPONENT  ANALYSIS 


EIGENVECTORS 


PRIN6 

PRIN7 

PRIN8 

PRIN9 

PRIN10 

XI 

0.065220 

-.185084 

0.318798 

0.339223 

0.515515 

X2 

-.008989 

0.074080 

0.043256 

0.131509 

0.070023 

X3 

0.124301 

0.152766 

-.233068 

-.344909 

-.179224 

X4 

-.169591 

0.216492 

0.224524 

0.232286 

-.131803 

X5 

0.090926 

0.384482 

0.140738 

-.019361 

0.565857 

X6 

-.534051 

-.048653 

-.138761 

-.085276 

0.297339 

X7 

-.147464 

0.156599 

0. 148703 

0.084184 

-.187037 

X8 

-.045080 

-.118376 

-.662186 

0.681261 

0.005320 

X9 

-.010021 

-.125001 

-.021633 

-.155073  ' 

-.068002 

X10 

-.138017 

0.200346 

0.019376 

0.158105 

-.325802 

Xll 

0.602462 

0.377237 

0.068697 

0.251417 

-.127454 

X12 

-.061335 

0.382822 

0.040467 

0.139263 

0.013158 

X13 

-.161653 

-.310900 

0.533749 

0. 288358 

-.325483 

X14 

0.471376 

-.509939 

0.003508 

-.032622 

0.055718 

PRIN11 

PRIN12 

PRIN13 

PRIN14 

XI 

0.121428 

0.028732 

-.209447 

0.528654 

X2 

0.012142 

0.003778 

0.006262 

-.028462 

X3 

-.055022 

-.053056 

-.019333 

0.109401 

X4 

-.374842 

-.601314 

0.275238 

0.275655 

X5 

0.131140 

0.082213 

0.421095 

-.441733 

X6 

0.050973 

0.011588 

-.037910 

-.024237 

X7 

-.314295 

0.784849 

0.240080 

0.284081 

X8 

-.083031 

0.043435 

0.110332 

-.134874 

X9 

-.009516 

-.005255 

-.005790 

0.020943 

X10 

0.838748 

0.014480 

0.092226 

0.196460 

Xll 

-.015001 

0.000861 

0.010199 

0.004646 

X12 

-.102931 

0.074573 

-.789099 

-.194181 

X13 

0.001998 

0.021967 

0.004358 

-.509974 

X14 

0.041114 

0.060967 

0.007364 

-.017769 
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DETAILED  DESIGN  PHASE:  Least  Squares  Method 

(using  Principal  Components) 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DF 

SQUARES 

SQUARE  F  VALUE 

PROB>F 

MODEL 

3 

2940.81945 

980.27315  .  6.616 

0.0342 

ERROR 

5 

740.88277 

148.17655 

C  TOTAL 

8 

3681.70222 

ROOT 

MSE 

12.17278 

R-SQUARE  0.7988 

DEP  MEAN 

24.25556 

ADJ  R-SQ  0.6780 

C.V. 

50.18553 

PARAMETER  ESTIMATES 

VARIABLE 

DF 

PARAMETER 

ESTIMATE 

STANDARD 

ERROR 

T  FOR  HO: 
PARAMETER-0 

INTERCEP 

1 

36.61996444 

7.48811514 

4.890 

PI 

1 

0 . 12930799 

0.03115062 

4 . 25i 

P2 

1 

-2.46877510 

0.78585547 

-3.142 

P3 

1 

-0.64069017 

0.32689299 

-1.960 

VARIABLE 

DF 

PROB  >  S  T  J 

INTERCEP 

1 

0.0045 

FI 

1 

0.0089 

P2 

1 

0.0256 

P3 

1 

0.1073 
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ACTUAL 


OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


ID 

4.08355 

29.1375 

1055.35 

6.94778 
22.1509 

39.8681 

87.6266 

100.724 

78.4993 
19.0778 

91.6872 

ID 

4.08355 

29.1376 

1055.35 

6.94778 
22.1505 

39.8681 

87.6266 

100.724 

78.4993 
19.0778 

91.6872 


ID 

4.08355 

29.1376 

1055.35 

6.94778 

22.1509 

39.8681 

87.6266 

100.724 

78.4993 

19.0778 

91.6872 


36.0000 

31.0000 

64.0000 

48.0000 

10.0000 

8.0000 

6.5000 

6.0000 

8.8000 


UPPER9  5  % 
MEAN 

55.1480 

51.6381 

93.9926 

45.4950 

32.7739 

29,9944 

27.3287 

40.4896 

22.6275 

34.2598 

43.0883 

STD  ERF 
RESIDUAL 

9.6849 
10. 1813 
0.6783 
10.5780 
10.2045 
7.9309 
11.2864 
8.8887 
7.5926 


PREDICT 

VALUE 

36.1923 
34.4870 
62.7505 
30.011? 
15.7.142 
6.2566 
15.6072 
19.1111 
-1.8305 
23 . 5373 
26.9036 
'.OVvER95% 
PREDICT 

-0.39.':l 

-1.1959 

18.5332 

-4.9005 

-19.9248 

-33.0192 

-17.8069 

-18,7855 

-41.5457 

-12.1034 

-8.3249 


STD  ERR 
PREDICT 

7.3742 
6.6722 
.12.1539 
6. 3234 
6.6366 
9.2345 
4.5599 
8.3167 
9. 5.147 
4.9104 
6.2962 
UPPE.R95% 
PREDICT 

72.7768 

70.1698 

107.0 

64.9235 

51.3532 

45.5324 

49.0213 

57.0076 

37.8847 

55.3780 

62.1322 


LOWER95% 

MEAN 

17.2367 
1/. 3359 
71.5085 
14.5280 
-1.3455 
-17.4811 
3.8057 
-2.2675 
-26. 2894 
9 . 01*\9 
10.7190 

RESIDUAL 

-0.1923 

-3.4870 

1.2495 

17.9885 

-5.7142 

1.7434 

-9.1072 

-13.1111 

10.6305 


SUM  OF  RESIDUALS  1.31672E-13 
SUM  OF  SQUARED  RESIDUALS  740.8828 
PREDICTED  RESID  SS  (PRESS)  164097.7 
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CODE  AND  UNIT  TEST  PHASE:  Stepwise  Selections 

STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 

WARNING:  2  OBSERVATIONS  DELETED  DUE  TO  MISSING  VALUES . 

NOTE:  SLENTRY  AND  SLSTAY  HAVE  BEEN  SET  TQ 
.15  FOR  THE  STEPWISE  TECHNIQUE. 


STEP  1 

VARIABLE  XI  ENTERED 

R  SQUARE  - 
C(P)  -  . 

0.67484762 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROR 

TOTAL 

1  2484.58798182 

7  1197.11424040 

8  3681.70222222 

2484.587982 

171.016320 

14.53 

0.0066 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

XI 

13.81140806 

5.69406301  1.49387^62 

2484,587982 

14.53 

0.0066 

BOUNDS  ON 

CONDITION  NUMBER: 

1# 

1 

STEP  2  VARIABLE  X9  ENTERED 


R  SQUARE  -  0.66541068 
C(P)  «»  . 


■  DF  SUM  OF  SQUARES  MEAN  SQUARE 


F  PROB>F 


REGRESSION 

ERROR 

TOTAL 


2  3186.18444000  1593.092220  19.29  0.0024 

6  495.51778222  02.586297 

8  3681.70222222 


B  VALUE  STD  ERROR  TYPE  II  S3 


F  PROB>F 


INTERCEPT 

XI 

X9 


11.91767775 
14.11517877 
-0 . 13589515 


3.07005978 

0.04662451 


1745.770281 

701.596458 


21.14  0.0037 
8.50  0.0268 


BOUNDS  ON  CONDITION  NUMBER:  8.745719,  34.98288 
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STEP  3 


VARIABLE  XI 1  ENTERED 


R  SQUARE  -  0.97400689 
C(P)  -  . 


DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PR03>F 

REGRESSION 

ERROR 

TOTAL 

3  3586.00334120 

5  95.69888102 

8  3681.70222222 

1195.334447 

19.139776 

62.45 

0.0002 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PR0B>F 

INTERCEPT 

XI 

X9 

XII 

17.61432258 

14.26567506  1.47832193 
-0.12192015  0.02265278 
-0.88959153  0.19463790 

1782.310925 

554.427691 

399.818901 

93.12 

28.97 

20.89 

0.0002 

0.0030 

0.0060 

BOUNDS  ON  CONDITION  NUMBER:  8.908018,  56.62403 


STEP  4  VARIABLE  X8  ENTERED  R  SQUARE  -  0.99235831 


C(P)  -  . 

DF  SUM  OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

ERROR 

TOTAL 

4  3653.56779471 

4  28.13442751 

8  3681.70222222 

913.3919487 

7.0336069 

129.86 

0.0002 

B  VALUE  STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

XI 

X8 

X9 

XII 

30.24108727 

14.41817104  0.89751810 
-2.32512285  0.75019785 
-0.06824032  0.02210312 
-1.04402729  0.12808083 

1815.147839 

67.564454 

67.042926 

467.340584 

258.07 

9.61 

9.53 

66.44 

0.0001 

0.0362 

0.0367 

0.0012 

BOUNDS  ON 

CONDITION  NUMBER:  23 

.07832,  193 

.2544 
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STEP  5 


VARIABLE  X6  ENTERED 


R  SQUARE  “  0.99716673 
C(P)  -  . 


DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

P 

PROB>F 

REGRESSION 

5  3671.27097173 

734.2541943 

211.17 

0.0005 

ERROR 

3 

10.43125049 

3.4770835 

TOTAL 

8  3681.70222222 

• 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

PROB>r 

INTERCEPT 

28.96925965 

XI 

15.63285056 

0 . 829465*8 

1235.079284 

355.21 

0.0003 

X6 

0.25403620 

0.11258423 

17.703177 

5.09 

0.1093 

X8 

-2.38561282 

0.52814664 

70.942420 

20.40 

0.0203 

X9 

-0.08100319 

0.01653809 

83.415842 

23.99 

0.0163 

Xll 

-1.19927580 

0.11332964 

389.373377 

111.98 

0.0018 

BOUNDS  ON 

CONDITION  NUMBER:  26 

.13549,  304 

.9335 

STEP  6 

VARIABLE  X18 

ENTERED 

R  SQUARE 

-  0.99965209 

C(P)  -  , 

DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

6  3680.42132510 

613.4035542 

957.77 

0.0010 

ERROR 

2 

1.28089412 

0.6404486 

TOTAL 

8  3681.70222222 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

28.34682750 

XI 

16.87615739 

0.43468544 

776.4463286 

1212.35 

0.0008 

X6 

0.20563632 

0.04998622 

10.8388523 

16.92 

0.0543 

X8 

-1.99036494 

0.24962427 

40.7170268 

63.58 

0.0154 

X9 

-0.11274143 

0.01099462 

67.3427958 

105.15 

0.0094 

Xll 

-1.10931860 

0.05409062 

269.6151332 

420.98 

0.0024 

X18 

-0.00386060 

0.00102136 

9 . 15C3534 

14.29 

0.0634 

ROUNDS  ON  CONDITION  NUMBER:  62.71203,  709.8545 
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STEP  7 

VARIABLE  X17 

ENTERED 

R  SQUARE 

-  0.99999735 

C(P)  -  . 

DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

7  3681.69247422 

525.9560677 

9999.99 

0.0001 

ERROR 

1 

0.00974801 

0.0097480 

TOTAL 

.  8  3681.70222222 

B  VALUE 

STD  ERROR 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

35.34086513 

XI 

16.55363291 

0.06613116 

610.7875117 

9999.99 

0.0001 

X6 

0.29622944 

0.01004828 

8.4720459 

869.11 

0.0216 

X8 

-2.00284334 

0.03081597 

41.1773272 

4224.18 

0.0098 

X9 

-0.10483743 

0.00152282 

46.2011223 

4739.55 

0.0092 

Xll 

-1.38270678 

0.02481132 

30.2744491 

3105.71 

0.0114 

X17 

-1.08045895 

0.09461672 

1.2711491 

130.40 

0.0556 

X18 

-0.00423198 

0.00013014 

10.3087888 

1057.53 

0.0196 

BOUNDS  ON  CONDITION  NUMBER:  79.0415,  1475.295 


STEP  8 

VARIABLE  X21 

ENTERED 

R  SQUARE 

-  1.00000000 

C(P>  -  . 

i 

DF  SUM 

OF  SQUARES 

MEAN  SQUARE 

F 

PROB>F 

REGRESSION 

8  3681.70222222 

460.2127778 

9999.99 

0.0001 

ERROR 

0 

0.00000000 

0.0000000 

TOTAL 

8  3681.70222222 

B  VALUE 

STD  ERROR' 

TYPE  II  SS 

F 

PROB>F 

INTERCEPT 

23.91179944 

XI 

16.40108872 

0 

94.85822032 

9999.99 

0.0001 

X6 

0.32541684 

0 

1.08333040 

9999.99 

0.0001 

X8 

-2.01511923 

0 

35.97475149 

9999.99 

0.0001 

X9 

-0.10108771 

0 

6.08156755 

9999.99 

0.0001 

Xll 

-1.46447109 

0 

2.86348417 

9999.99 

0.0001 

X17 

-1.33068597 

0 

0.24119088 

9999.99 

0.0001 

X18 

-0.00380003 

0 

0.69167302 

9999.99 

0.0001 

X21 

12.99647065 

0 

0.00974801 

9999.99 

0.0001 

BOUNDS  ON  CONDITION  NUMBER:  558.2853,  13148.58 


NO  OTHER  VARIABLES  MET  THE  0.1500  SIGNIFICANCE  LEVEL  FOR  ENTRY 
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SUMMARY  OF  STEPWISE  REGRESSION  PROCEDURE  FOR  DEPENDENT  VARIABLE  Y 


VARIABLE 

NUMBER 

PARTIAL 

MODEL 

STEP 

ENTERED  REMOVED 

IN 

R**2 

R**2 

1 

XI 

1 

0.6748 

0.6748 

2 

X9 

2 

0.1906 

0.8654 

3 

Xll 

3 

0.1086 

0.9740 

4 

X8 

4 

0.0184 

0.9924 

5 

X6 

5 

0.0048 

0.9972 

6 

X18 

6 

0.0025 

0.9997 

7 

X17 

7 

0.0003 

1.0000 

8 

X21 

8 

0.0000 

1.0000 

VARIABLE 


STEP 

ENTERED  REMOVED 

F 

1 

XI 

14.5284 

2 

X9 

8.4953 

3 

Xll 

20.8894 

4 

X8 

9.6059 

5 

X6 

5.0914 

6 

X18 

14.2874 

7 

X17 

130.4009 

8 

X21 

9999.9999 

PROB>F 

0.0066 
0.0268 
0.0060 
0.0362 
0.1093 
0 1  0634 
0.0556 
0.0001 


C(P) 
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CODE  AND  UNIT  TEST  PHASE:  Stepwise  Regression  Analysis 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 
SUM  OF  MEAN 

SOURCE  DF  SQUARES  SQUARE  F  VALUE  PROB>P 

MODEL  8  3681.70222  460.21278 

ERROR  0  9 . 95121E-12 

C  TOTAL  ‘  8  3681.70222 


ROOT  MSE  .  R-SQUARE  1.0000 

DEP  MEAN  24.25556  ADJ  R-SQ 

C.V. 

PARAMETER  ESTIMATES 


VARIABLE 

DF 

PARAMETER 

ESTIMATE 

INTERCEP 

1 

23.91179944 

XI 

1 

16.40108872 

X6 

1 

0.32541684 

X8 

1 

-2.01511923 

X9 

1 

-0.10108771 

Xll 

1 

-1.46447110 

X17 

1 

-1.33068597 

X18 

1 

-0.003800033 

2C21 

1 

12.99647065 

VARIABLE 

DF 

PROB  >  | T  j 

INTERCEP 

1 

• 

XI 

1 

• 

X6 

1 

• 

X8 

1 

e 

X9 

1 

• 

Xll 

1 

• 

X17 

1 

• 

X18 

1 

• 

X21 

1 

• 

STANDARD  T  FOR  HO: 

ERROR  PARAMETER" 0 
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CODE  AND  UNIT  TEST  PHASE:  Principal  Component  Analysis 


PRINCIPAL  COMPONENT  ANALYSIS 
11  OBSERVATIONS 


22 

VARIABLES 

SIMPLE 

STATISTICS 

XI 

X2 

X3 

X4 

MEAN 

ST  DEV 

2.87712 

4.52637 

105.297 

243.706 

20.0766 

26.7754 

2.83031 

3.99664 

X6 

X7 

X8 

X9 

MEAN 

ST  DEV 

7.11785 

8.39976 

2.11232 

2.56330 

8.02100 

4.62116 

90.035 

183.950 

Xll 

X12 

X15 

XI? 

MEAN 

ST  DEV 

7.29776 

8.21082 

2.43642 

3.64045 

70585.5 

90661.0 

P.45455 

2.33939 

X19 

X20 

X21 

X22 

MEAN 

ST  DEV 

102.455 

55.682 

21.1818 

17.9990 

0.990909 

0.020226 

2.36364 

0.80904 

X24 

X25 

MEAN 

ST  DEV 

2.74685 

3.34931 

22.6800 

15.8044 

X5 

2.59865 

3.05383 


X10 

2.05379 

2.65265 


X18 

419.000 

494.511 


X23 

2.52727 

2.77708 
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PRINCIPAL  COMPONENT  ANALYSIS 


COVARIANCES 


XI 

X2 

X3 

X4 

X5 

X6 

XI 

20.488 

507.627 

76.7436 

-6.88532 

4.59364 

-13.2205 

X2 

507.627 

59392.6 

6000.55 

-9.22677 

26.4418 

5.33685 

X3 

76.7436 

6000.55 

716.922 

11.1813 

23.2099 

-.195098 

X4 

-6.88532 

-9.22677 

11.1813 

15.9731 

5.96383 

15.8104 

X5 

4.59364 

26.4418 

23.2099 

5.96383 

9.32588 

-6.66383 

X6 

-13.2205 

5.33685 

-.195098 

15.8104 

-6.66383 

70.556 

X7 

-.495505 

327.701 

41.0176 

8.56419 

3.46024 

9.12385 

X8 

15.039 

949.037 

108.598 

-1.976 

3.52059 

-6.63936 

X9 

391.763 

44639.8 

4421.27 

-34.1497 

12.0465 

7.76958 

X10 

-2.19708 

215.614 

30.1225 

9.80594 

4.3422 

8.10716 

Xll 

0.731235 

785.26 

46.8588 

-6.66139 

-8.98976 

34.3681 

X.12 

-3.84745 

139.957 

25.7475 

13.3912 

7.72505 

5.65109 

X13 

-85891.3 

-6993472 

-1125733 

-173297 

-121470 

-208659 

X17 

-2.24013 

-67.8237 

-2.55027 

3.42?44 

0.814652 

-.740503 

X18 

-310.767 

-34904.1 

-5612.36 

-1011.46 

-665.931 

-1412.74 

X19 

32.6381 

-3627.07 

7.38965 

105.698 

65.6658 

123.696 

X20 

-7.72981 

-467.318 

35.1436 

43.4028 

37.4768 

-7.76181 

X21 

-0.04723 

1.00458 

.0225336 

0.02798 

-.011351 

.0709393 

X22 

1.63621 

31.2285 

0.970624 

-2.6418 

-1.18823 

-.977816 

X23 

4.00557 

630.372 

67.4554 

3.63113 

2.43762 

1.78892 

X24 

-3.33469 

49.5614 

21.6735 

12.3389 

7.18182 

6.57276 

X25 

29.9385 

2403.47 

323.785 

25.635 

11.0381 

46.6303 

E-65 


X7 

XJ? 

X9 

X10 

Xll 

X12 

XI 

-.495505 

15.039 

391.763 

-2.19708 

0.731235 

-3.84745 

X2 

327.701 

949.037 

44639.8 

215.614 

785.26 

139.957 

X3 

41.0176 

108.598 

4421.27 

30.1225 

46.8533 

25.7475 

X4 

8.56419 

-1.976 

-34.1497 

9.80594 

-6.66139 

13.3912 

X5 

3.46024 

3.53059 

12.046*" 

*•3422 

-8.98976 

7.72505 

X« 

9.12385 

-6.63936 

7.76*r  3 

S.  10716 

34.3681 

5.65109 

X7 

6.5705 

4.27394 

2 3 C  4/3' 

6.54925 

0.495098  - 

7.941 

X8 

4.27394 

21.3551 

721.  M  4 

2.82043 

6.72665 

1.76349 

X9 

230.478 

721.786 

33837.8 

144.485 

662.567 

83.1111 

X10 

6.54925 

2.82043 

144.485 

7.03654 

-2.72536 

9.17345 

Xll 

0.495098 

6.72665 

662.567 

-2.72536 

67.4176 

-7.23447 

X12 

7.941 

1.76349 

83.1111 

9.17345 

-7.23447 

13.2529 

X15 

-139309 

-161883 

-5188362 

-137945 

115772 

-165204 

X17 

1.54663 

0.804153 

-62.6251 

2.38462 

-12.4703 

2.99725 

X18 

-782.588 

-635.836 

-26117.9 

-767.761 

187>  i  637 

-938.115 

X19 

46.1192 

-1.10482 

-3110.33 

57.2551 

-206.265 

71.0751 

X20 

20.9603 

14.0138 

-413.461 

30.1997 

-72.7273 

47.5335 

X21 

.0180623 

-.009843 

0.799326 

.0186701 

.0701341 

.0221496 

X22 

-1.17045 

0.360237 

28.9647 

-1.60379 

2.74265 

-2.43871 

X23 

5.48843 

9.75502 

466.099 

4.68082 

6.06845 

5.06006 

X24 

6.99544 

0.741977 

8.56358 

8.19207 

-8.53385 

11.7909 

X25 

27.366 

51.4449 

1758.18 

24,0727 

12.0449 

22.1815 
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PRINCIPAL  COMPONENT  ANALYSIS 


COVARIANCES 


X15 

X17 

X18 

X19 

X20 

X21 

XI 

**85891 . 3 

-2.24013 

-310.767 

32.6381 

-7.72981 

-0.04723 

X2 

-6993472 

-67.8237 

-34904.1 

-3627.07 

-467.318 

1.00458 

X3 

-1125733 

-2.55027 

-5612.36 

7.38965 

35.1436 

.0225336 

X4 

-173297 

3.42244 

-1011.46 

105.698 

43.4028 

0.02798 

X5 

-121470 

0.814652 

-665.931 

65.6658 

37.4768 

-.011351 

X6 

-208659 

-.740503 

-1412.74' 

123.696 

-7.76181 

.0709393 

X7 

-139309 

1.54663 

-782.588 

46.1192 

20.9603 

.0180623 

X8 

-161883 

0.804153 

-635.836 

-1.10482 

14.0138 

-.009843 

X9 

-5188362 

-62.6251 

-26117.9 

-3110.33 

-413.461 

0.799326 

X10 

-137945 

2.38462 

-767.761 

57.2551 

30.1997 

.0186701 

Xll 

115772 

-12.4703 

183.637 

-206.265 

-72.7273 

.0701341 

X12 

-165204 

2.99725 

-938.115 

71.0751 

47.5335 

.0221496 

X15 

8 • 2E+09 

-100343 

43466853 

-1027794 

-886084 

-531.93 

X17 

-100343 

5.47273 

-421.9 

47.1727 

30.3091 

.0045455 

X18 

43466853 

-421.9 

244541 

-3143.5 

-4314.1 

-4.18 

XI 9 

-1027794 

47,1727 

-3143.5 

3100.47 

451.909 

-.620455 

X20 

-886584 

30.3091 

-4314.1 

451.909 

323.964 

.0318182 

X21 

-531.93 

.0045453 

-4.18 

-.620455 

.0318182 

4.1E-04 

X22 

29464.8 

-1.18182 

153.7 

-13.9818 

-12.0727 

-.006364 

X23 

-110036 

-.103636 

-585.29 

-15.2036 

7.29455 

.0152727 

X24 

-155027 

3.02282 

-866.958 

89.9873 

45.7 

.0147347 

X25 

-964771 

11.8092 

-4723.07 

242.107 

93.5026 

.0678547 
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X22 


X23 


X24 


X25 


XI 

1.63621 

4.00557 

-3.33469 

X2 

31.2285 

630.372 

49.5614 

X3 

0.970624 

67.4554 

21.6735 

X4 

-2.6418 

3.63113 

12.3389 

X5 

-1.18823 

2.43762 

7.18182 

X6 

-.977816 

1.7889? 

6.57276 

X7 

-1.17045 

5.48843 

6.99544 

X8 

0.360237 

9.75502 

0.741977 

X9 

28.9647 

466.099 

8.56358 

X10 

-1.60379 

4.68082 

8.19207 

Xll 

2.74265 

6.06845 

-8.53385 

X12 

-2.43871 

5.06006 

11.7909 

X1S 

29464.8 

-110036 

-155027 

X17 

-1.18182 

-.103636 

3.02282 

X18 

153.7 

-585.29 

-866.958 

X19 

-13.9818 

-15.2036 

89.9873 

X20 

-12.0727 

7.29455 

45.7 

X21 

-.006364 

.0152727 

.0147347 

X22 

0.654545 

-.330909 

-2.29616 

X23 

-.330909 

,  7.71218 

3.83036 

X24 

-2.29616 

3.83036 

11.2179 

X25 

-4.01265 

29.4533 

21.0877 

29.9385 

2403.47 

323.785 

25.635 

11.0381 

46.6303 

27.366 

51.4449 

1758.18 

24.0727 

12.0449 

22.1815 

-964771 

11.8092 

-4723.07 

242.107 

93.5026 

.0678547 

-4.01265 

29.4533 

21.0877 

249.778 
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PRINCIPAL  COMPONENT  ANALYSIS 


TOTAL  VARIANCE-8219758114 


EIGENVALUE  DIFFERENCE  PROPORTION 


PRIN1 

8.2E+09 

PRIN2 

84900 

PRIN3 

15138.8 

PRIN4 

2189.6 

PRIN5 

225.454 

PRIN6 

159.679 

PRIN7 

90.0073 

PRIN8 

73.3826 

FRIN9 

19.2198 

PRIN10 

9.72102 

PRIN11 

0 

FRXN12 

0 

FRIN13 

0 

PRIN14 

0 

PRIN15 

0 

PRIN16 

0 

PRIN17 

0 

PRIN18 

0 

PRIN19 

0 

PRIN20 

0 

PRIN21 

0 

FRIN22 

0 

8.2E+09 

0.999987 

69761.1 

0.000010 

12949.2 

0.000002 

1964.14 

0.000000 

65.7749 

0.000000 

69.6715 

0.000000 

16.6248 

0.000000 

54.1628 

0.000000 

9.49877 

0.000000 

9.72102 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

0 

0.000000 

• 

0.000000 

CUMULATIVE 

0.99999 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 

1.00000 
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PRINCIPAL  COMPONENT  ANALYSIS 


EIGENVECTORS 


PRIN1 

PRIN2 

PRIN3 

PRIN4 

PRIN5 

PRXN6 

XI 

-.000010 

0.006547 

0.008942 

0.015709 

-.056049 

-.027712 

X2 

-.000851 

0.792770 

-.005018 

0.125958 

0.290333 

-.394896 

X3 

-.000137 

0.074101 

0.013055 

0.136146 

0.177494 

-.283701 

X4 

-.000021 

-.002599 

-.004742 

0.041235 

0.080102 

0.033727 

X5 

-.000015 

-.001226 

-.000662 

0.022608 

0.095339 

0.062615 

X6 

-.000025 

-.002694 

-.018707 

0.063500 

-.286712 

0.012262 

X7 

-.000017 

0.002939 

-.002872 

0.027039 

0.039931 

0.012168 

X8 

-.000020 

0.012140 

0.012796 

.  0.006392 

0.010191 

0.068581 

X9 

-.000631 

0.598875 

-.025286 

-.062719 

-.394262 

0.584968 

X10 

-.000017 

0.001283 

-.002044 

0.026875 

0.067131 

0.033779 

Xll 

0.000014 

0.013522 

-.031911 

-.011518 

-.272706 

0.155914 

X12 

-.000020 

-.000226 

-.003417 

0.030181 

0.127467 

0.068530 

X15 

0.999985 

0.000381 

-.005163 

0.001263 

0.000170 

0.000130 

X17 

-.000012 

-.002327 

0.007790 

-.000286 

0.060955 

0.020186 

X18 

0.00*288 

0.032807 

0.978640 

-.195482 

-.016331 

-.011999 

X19 

-.000125 

-.070327 

0.195277 

0.947814 

-.146752 

0.081288 

X20 

-.000108 

-.018550 

0.031153 

0.074918 

0.698184 

C. 606836 

X21 

-.000000 

0.000008 

-.000099 

-.000159 

0.  COCK!  S':. 

0.000162 

X22 

0.000004 

0.000876 

-.000384 

-.002532 

-. 0340  ;l 

-.012485 

X23 

-.000013 

0.007837 

-.001381 

0.011288 

0.038796 

0.007426 

X24 

-.000019 

-.001483 

-.001897 

0.034407 

0.120677 

0.029016 

X25 

-.000117 

0.023148 

0.023930 

0.095601 

-.029662 

-.062825 
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PRIN7 

PRIN8 

PRIN9 

PRIN10 

PRIN11 

PRIN12 

XI 

-.293858 

0.275821 

-.025936 

0.014946 

0.048740 

-.348375 

X2 

0.156175 

-.179663 

0.062062 

-.093238 

-.025203 

0.014341 

X3 

-.267890 

0.565052 

0.420062 

0.125267 

-.031861 

-.029553 

X4 

0.180554 

-.109308 

-.111397 

0.454105 

-.192343 

0.075215 

X5 

-.135989 

0.058072 

0.063965 

0.407151 

-.150030 

0.219978 

X6 

0.567977 

-.094870 

0.336352 

-.094909 

-.011000 

-.089616 

X7 

0.083063 

-.070265 

-.049460 

0.186226 

-.082482 

0.300076 

XB 

-.077519 

0.151142 

0.034662 

-.085038 

0.041736 

0.409439 

X9 

-.200525 

0.123037 

-.131569 

0.092745 

0.037645 

-.021297 

X10 

0.094753 

-.076646 

-.049870 

0.215596 

-.098653 

-.019585 

XI 1 

0.237169 

0.146081 

0.614340 

0.052314 

-.040643 

0.012401 

X12 

0.084185 

-.097099 

-.033303 

0.452755 

-.194558 

-.285452 

X15 

-.000084 

0.000099 

-.000199 

-.000126 

0.000011 

0.000013 

X17 

0.057001 

-.035248 

-.174552 

-.320315 

0.110811 

-.133557 

X18 

0.025319 

-.005396 

0.033735 

0.024263 

-.001820 

-.003126 

X19 

-.099915 

-.108883 

-.045596 

-.042101 

0.001505 

0.004463 

X20 

0.141469 

0.093127 

0.161746 

-.225223 

-.048940 

-.025880 

X21 

0.001300 

-.000066 

-.000282 

-.000126 

-.000093 

0.334952 

X22 

-.040800 

0.002275 

0.013730 

-.052786 

0.026725 

0.546899 

X23 

0.022714 

-.030572 

0.013111 

0.137401 

-.060137 

-.210299 

X24 

0.001935 

-.061950 

0.063160 

0.333376 

0.926134 

-.000054 

X25 

0.524962 

0.664756  -.472468  0.034461  0.020334 

PRINCIPAL  COMPONENT  ANALYSIS 

EIGENVECTORS 

0.035530 

PRIN13 

PRIN14 

PRIN15 

PRIN16 

PRIN17 

PRIN18 

XI 

0.199128 

-.359479 

0.391893 

-.136960 

0.390616 

-.196401 

X2 

0.005427 

-.199781 

-.003183 

-.016528 

0.027346 

-.014632 

X3 

-.008700 

0.490104 

0.008092 

0.067065 

-.027554 

0.041603 

X4 

-.022381 

0. 180401 

-.260259 

-.007030 

0.735973 

0.000068 

X5 

-.095413 

-.248823 

0.111701 

-.681884 

-.243781 

0.148275 

X6 

0.041258 

0.287062 

0.507988 

-.286705 

0.118915 

0.007171 

X7 

-.173016 

0.181651 

0.076335 

0.003010 

-.240268 

-.697337 

X8 

-.297886 

-.030281 

0.284005 

0.299489 

0.195084 

0.109361 

X9 

-.004497 

0.214741 

0.004291 

0.004041 

-.030021 

0.017103 

XIO 

0.037686 

0.083519 

0.084083 

0.105474 

-.164224 

0.633405 

Xll 

0.007912 

-.401870 

-.377370 

0.191306 

-.057342 

-.031783 

X12 

0,228759 

-.087482 

0.364731 

0.492725 

-.251713 

-.094534 

X15 

-.000013 

-.000004 

0.000016 

0.000001 

0.000002 

0.000008 

X17 

0.044917 

0.246077 

0.002049 

-.055759 

-.088485 

0.066265 

X18 

0.003053 

0.005759 

0.000606 

-.002667 

0.000074 

-.001606 

XI 9 

-.008090 

-.040399 

-.045695 

0.018790 

-.018198 

0.003918 

X20 

0.047828 

-.025298 

0.026223 

-.057862 

0.065408 

-.047505 

X21 

-.268063 

-.077573 

0.275962 

0.175039 

0.048879 

0.135981 

X22 

0.829799 

0.039323 

0.022126 

0.011990 

0.023361 

0.005436 

X23 

0.106060 

0.262388 

-.229525 

-.106444 

-.121024 

-*.05068* 

X24 

0.000030 

0.000775 

0.000021 

0.000048 

-.000068 

0.000024 

X25 

0.03,9472 

-.135545 

-.083447 

-.010290 

-.101755 

0.003187 

E-71 


PRIN19 

PRIN20 

PRIN21 

FRIN22 

XI 

0.312574 

0.192387 

0.164207 

0.120852 

X2 

0.013817 

0.033314 

-.010028 

0.052385 

X3 

-.042890 

-.019354 

0.032367 

-.167744 

X4 

-.088892 

0.142406 

0.058971 

-.103824 

X5 

-.257313 

0.163611 

-.030508 

0.008069 

X6 

-.022148 

-.084717 

-.069407 

0.018173 

X7 

0.424821 

0.228990 

0.049723 

-.060920 

-.112328 

0.282543 

-.425997 

0.451916 

-.019425 

-.054921 

0.011163 

-.071648 

X10 

0.649457 

0.205211 

0.003963 

-.031687 

Xll 

0.024771 

0.277025 

0.13452/ 

0.011138 

X12 

-.350791 

0.050171 

-.036486 

-.034579 

X15 

-.000016 

-.000024 

-.000027 

0.000020 

X17 

-.274292 

0.764548 

0.286583 

-.091256 

X18 

0.002874 

0.003550 

0.005124 

-.005044 

X19 

-.011927 

-.008882 

0.000199 

0.002506 

X20 

0.063823 

-.103640 

0.010887 

-.017095 

X21 

-.083387 

-.217809 

0.791683 

0.033157 

X22 

-.008422 

0.006488 

0.040242 

0.034118 

X23 

0.012418 

-.058269 

0.220111 

0.846512 

X24 

-.000033 

0.000167 

0.000161 

-.000250 

X25 

-.017378 

-.066171 

-.006465 

0.021055 
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CODE  AND  UNIT  TEST  PHASE:  Least  Squares  Method 

(using  Principal  Components) 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF  MEAN 


SOURCE 

DF 

SQUARES 

SQUARE 

F  VALUE 

PR0B>F 

MODEL 

1 

275.45981 

275.45981 

0.566 

0.4764 

ERROR 

7 

3406.24241 

486.60606 

C  TOTAL 

8 

3681.70222 

ROOT 

MSE 

22.05915 

R-SQUARE 

0.0748 

DEP  1 

MEAN 

24.25556 

ADJ  R-SQ 

-0.0574 

C.V. 

90.94473 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER-0 

INTERCEP 

1 

38.02688694 

19.72530460 

1.928 

PI 

1 

-2.58584501 

3.43686223 

-0.752 

VARIABLE 

DF 

PROB  >  | T | 

INTERCEP 

1 

0.0952 

PI 

1 

0.4764 

OBS 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWERS 5 % 
MEAN 

1 

5.76979 

36.0000 

23.1071 

7.5098 

5.3491 

2 

6.15126 

31.0000 

22.1207 

7.8815 

3.4836 

3 

3.13332 

64.0000 

29.9246 

10.5281 

5.0295 

4 

5.2445 

48.0000 

24.4654 

7.3583 

7.0656 

5 

2.27999 

10.0000 

32.1312 

12.7921 

1.8825 

6 

2.27047 

8.0000 

32.1558 

12.8189 

1.8438 

7 

7.61732 

6.5000 

18.3297 

10.7750 

-7.1493 

8 

7.55941 

6.0000 

18.4794 

10.6304 

-6.6577 

9 

7.90488 

8.8000 

17.5861 

11.5172 

•9.6479 

10 

5.91787 

• 

22.7242 

7.6205 

4.6830 

11 

10.373 

• 

11.2038 

18.8412 

-33.3490 
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OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


OBS 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


ID 

5.76979 

6.15126 

3.13332 

5.2445 

2.27999 

2.27047 

7.61732 

7.55941 

7.90488 

5.91787 

10.373 


ID 

5.76979 
6.15126 
3.13332 
5.2445 
2.27999 
2.27047 
7.61732 
7.55941 
7.90488 
5 .91787 
10.373 


UPPER95% 

MEAN 

40.8651 

40.7577 

54.8198 

41.8653 

62.3798 

62.4679 

43.8087 

43.6165 

44.8201 

40.7653 

55.7566 

STD  ERR 
RESIDUAL 

20.7415 

20.6031 

19.3847 

20.7937 

17.9714 

17.9522 

19.2485 

19.3288 

18.8138 


LOWER95% 

PREDICT 

-31.9948 

-33.2708 

-27.8737 

-30.5221 

-28.1669 

-28.1741 

-39.7225 

-39.4235 

-41.2575 

-32.4696 

-57.3952 


UPPERS 5% 
PREDICT 

78.2090 

77.5121 

87.7229 

79.4530 

92.4292 

92.4857 

76.3819 

76.3823 

76.4297 

77.9180 

79.8028 


RESIDUAL 

12.8929 

8.8793 

34.0754 

23.5346 

-22.1312 

-24.1558 

-11.8297 

-12.4794 

-8.7861 


SUM  OF  RESIDUALS  9.14824E-14 
SUM  OF  SQUARED  RESIDUALS  3406.242 
PREDICTED  RESID  SS  (PRESS)  6058.221 


Regression  Models  Using  Principal  Components  Analysis 
for  Highly  Correlated  Variables 

PRINCIPAL  COMPONENT  ANALYSIS 

11  OBSERVATIONS 
7  VARIABLES 


SIMPLE  STATISTICS 


XI 

X2 

X3 

X8 

MEAN 

2.S7712 

105.297 

20.0766 

8.02100 

ST  DEV 

4.52637 

243.706 

26.7754 

4.62116 

X9 

X23 

X25 

MEAN 

90.035 

2.52727 

22.6800 

ST  DEV 

183.950 

2.77708 

15.8044 

' 

COVARIANCES 


XI 

X2 

X3 

X8 

XI 

20.48802 

507.6274 

76.7436 

15.03902 

X2 

507.6274 

59392.56 

6000.551 

949.0371 

X3 

76.7436 

6000.551 

716.922 

108.5977 

X8 

15.03902 

949.0371 

108.5977 

21.35512 

X9 

391.7628 

44639.85 

4421.269 

721.7865 

X23 

4.005569 

630.3725 

67.45542 

9.755015 

X25 

29.93848 

2403.468 

323.7854 

51.44488 

X9 

X23 

X25 

XI 

391.7628 

4.005569 

29.93848 

X2 

44639.85 

630.3725 

2403.468 

X3 

4421.269 

67.45542 

323.7854 

X8 

721.7865 

9.755015 

51.44488 

X9 

33837.77 

466.0989 

1758.177 

X23 

466.0989 

7.712182 

29.45329 

X25 

1758.177 

29.45329 

249.7776 

E-75 


PRINCIPAL  COMPONENT  ANALYSIS 
TOTAL  VARIANCE-94246.59 


CUMULATIVE 

EIGENVALUE 

DIFFERENCE 

PROPORTION 

PRIN1 

0.99492 

93767.7 

93474.0 

0.994919 

PRIN2 

0.99803 

293.6 

163.8 

0.003116 

PRIN3 

0.99941 

129.8 

81.3 

0.001377 

PRIN4 

0.99993 

48.5 

43.9 

0.000515 

PRIN5 

0.99998 

4.6 

2.5 

0.000049 

PRIN6 

1.00000 

2.1 

1.8 

0.000022 

PRIN7 

1.00000 

0.3 

• 

'0.000003 

EIGENVECTORS 

PRIN1 

PR1N2 

PRIN3 

PRIN4 

XI 

0.006891 

0.053822 

0.138704 

0.458959 

X2 

0.795466 

0.322656 

-.460802 

-.164212 

X3 

0.079926 

0.560871 

0.141991 

0.674916 

X8 

0.012782 

0.040068 

0.121577 

>  0.146618 

X9 

0.599617 

-.533522 

0.553546 

0.147258 

X23 

0.008398 

0.031659 

-.015624 

-.029001 

X25 

0.032006 

0.539598 

0.653317 

-.512696 

PR1N5 

PRIN6 

PRIN7 

XI 

0.658582 

-.466968 

0.339683 

X2 

0.151586 

-.020767 

0.020326 

X3 

-.421264 

0.072819 

-.143518 

X8 

0.500471 

0.840668 

-.068991 

X9 

-.164496 

0.006469 

-.024806 

X23 

-.280461 

0.248635 

0.925940 

X25 

0.098187 

-.087309 

0.029412 

E-76 


PRINCIPAL  COMPONENT  ANALYSIS 


11  OBSERVATIONS 


10 

VARIABLES 

SIMPLE 

STATISTICS 

X4 

X5 

X7 

X10 

MEAN 

ST  DEV 

2.83031 

3.99664 

2.59865 

3.05383 

2.11232 

2.56330 

2.05379 

2.65265 

X18 

X19 

X20 

X22 

MEAN 

ST  DEV 

419.000 

494.511 

102.455 

55.682 

21.1818 

17.9990 

2.36364 

0.80904 

COVARIANCES 


X4 

X5 

X7 

X10 

X4 

15.9731 

5.96383 

8.56419 

9.80594 

X5 

5.96383 

9.32588 

3 . 48024 

4.3422 

X7 

8.56419 

3.46024 

6.5705 

6.54925 

X10 

9.80594 

4.3422 

6.54925 

7.03654 

X12 

13.3912 

7.72505 

7.941 

9.17345 

X18 

-1011.46 

-665.931 

-782.588 

-767.761 

X19 

105.698 

65.6658 

46.1192 

57.2551 

X20 

43.4028 

37.4768 

20.9603 

30.1997 

X22 

-2.6418 

-1.18823 

-1.17045 

-1.60379 

X24 

12.3389 

7.18182 

6.99544 

8.19207 

X18 

X19 

X2  0 

X22 

X4 

-1011.46 

105.698 

43.4028 

-2.6418 

X5 

-665.931 

65.6658 

37.4768 

-1.18823 

X7 

-782.588 

46.1192 

20.9603 

-1.17045 

X10 

-767.761 

57.2551 

30.1997 

-1.60379 

X12 

-938.115 

71.0751 

47.5335 

-2.43871 

X18 

244541 

-3143.5 

-4314.1 

153.7 

X19 

-3143.5 

3100.47 

451.909 

-13.9818 

X20 

-4314.1 

451.909 

323.964 

-12.0727 

X22 

153.7 

-13.9818 

-12.0727 

0.654545 

X24 

-866.958 

89.9873 

45.7 

-2.29616 

E-77 


X12 

2.43642 

3.64045 


X24 

2.74685 

3.34931 


X12 

13.3912 

7.72505 

7.941 

9.17345 

13.2529 

-938.115 

71.0751 

47.5335 

-2.43871 

11.7909 


X24 

12.3389 

7.18182 

6.99544 

8.19207 

11.7909 

-866.958 

89.9873 

45.7 

-2.29616 

11.2179 


PRINCIPAL  COMPONENT  ANALYSIS 


TOTAL  VARIANCE-248029.9 


EIGENVALUE 

DIFFERENCE 

PROPORTION 

CUMULATIVE 

PRIN1 

244676 

241553 

0.986480 

0.98648 

PRIN2 

3123 

2921 

0.012591 

0.99907 

PRIN3 

202 

180 

0.000813 

0.99988 

PRIN4 

22 

17 

0.000088 

0.99997 

PRIN5 

5 

4 

0.000020 

0.99999 

PRIN6 

1 

1 

0.000006 

1.00000 

PRIN7 

0 

0 

0.000001 

1.00000 

PRIN8 

0 

0 

0.000000 

1.00000 

PRIN9 

0 

0 

0.000000 

1.00000 

PRINIO 

0 

• 

0.000000 

1.00000 

EIGENVECTORS 


PRIN1 

PRIN2 

PRIN3 

PRIN4 

PRIN5 

X4 

-.004142 

0.030741 

0.074788 

0.557404 

-.255778 

X5 

-.002728 

0.019341 

0.093433 

0.086419 

0.933384 

X7 

-.003202 

0.011849 

0.017808 

0.359227 

-.076413 

XI 0 

-.003143 

0.015895 

0.057780 

0.362158 

-.098286 

X12 

-.003841 

0.020303 

0.124832 

0.480359 

0.144420 

X18 

0.999722 

0.015776 

0.016865 

0.004420 

0.000214 

X19 

-.013048 

0.988986 

-.145044 

-.023291 

-.000806 

X20 

-.017678 

0.137151 

0.965536 

-.190685 

-.090993 

X22 

0.000630 

-.004264 

-.039851 

-.062573 

0.079428 

X24 

-.003551 

0.026534 

0.106152 

0.387360 

0.111473 

PRIN6 

PRIN7 

PRIN8 

PRIN9 

PRINIO 

X4 

-.580518 

0.358151 

-.282895 

0.259850 

-.067073 

X5 

-.059107 

0.189811 

-.259442 

0.043326 

0 . 059209 

X7 

0.679620 

0.062402 

-.365802 

0.049403 

-.512465 

X10 

0.392905 

0.100794 

-.033061 

0.024990 

0.830259 

X12 

0.022076 

0.164348 

0.739272 

-.348577 

-.191995 

X18 

0.000998 

0.000412 

-.000080 

0.000605 

-.000169 

X19 

0.003909 

-.000096 

0.008653 

-.008030 

-.000077 

X20 

0.033509 

0.026429 

-.018878 

0.038192 

-.018449 

X22 

0.152950 

0.083966 

0.412597 

0.885806 

-.053252 

X24 

-.135011 

-.887066 

-.023766 

0.140904 

0.002716 

E-78 


PRINCIPAL  COMPONENT  ANALYSIS 

11  OBSERVATIONS 
2  VARIABLES 


SIMPLE  STATISTICS 

Xll  X17 

MEAN  7.29776  5.45455 

ST  DEV  8.21082  2.33939 


COVARIANCES 


Xll 


X17 


Xll  67.41757  -12.4703 

X17  -12.4703  5.472727 


TOTAL  VARIANCE-72.89029 


CUMULATIVE 

EIGENVALUE 

DIFFERENCE 

PROPORTION 

PRIN1 

0.95807 

PRIN2 

1.00000 

69.8338 

3.0565 

66.7772 

• 

0.958067 

0.041933 

EIGENVECTORS 

PRIN1 

PRIN2 

Xll 

X17 

0.981742 

-.190218 

0.190218 

0.981742 

E-79 


Regression  Model:  using  the  first  Principal  component 

of  each  highly  correlated  set  • >f  variable*. 


DSP  VARIABLE:  Y 


ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DP 

SQUARES 

SQUARE 

F  VALUE 

PROB>F 

MODEL 

5 

2744.60060 

548.92012 

1.757 

0.3408 

ERROR 

3 

937.10162 

312.36721 

C  TOTAL 

8 

3681.70222 

ROOT  MSE 

17.67391  R-SQUARE  0.7455 

DEP  MEAN 

24.25556  ADJ 

R-SQ  0.3213 

C.V. 

72.86542 

PARAMETER 

ESTIMATES 

« 

PARAMETER 

STANDARD  T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR  PARAMETER- 0 

INTERCEP 

1 

303.72064 

432.56932  0.702 

PI 

1 

0.04912381 

0.02107148  2.331 

P2 

1 

0.01548917 

0.01901178  '  0.815 

P3 

1 

0.04067242 

0.06340874  0.641 

P4 

1 

-0.64732832 

0.89593575  -0.723 

P5 

1 

-290.92605 

439.15510  -0.662 

VARIABLE 

DF 

PROB  >  J  T | 

INTERCEP 

1 

0.5332 

PI 

1 

0.1020 

P2 

1 

0.4749 

P3 

1 

0.5669 

P4 

1 

0.5222 

P5 

1 

0.5550 

E-80 


OBS 


OBS 


OBS 


PREDICT 

STD  ERR 

LOWER95% 

ID 

ACTUAL 

VALUE 

PREDICT 

MEAN 

1 

4.1398 

36.0000 

36.0000 

17.6739 

-20.2472 

2 

29.2134 

31.0000 

30.4031 

17.6639 

-25.8124 

3 

1057.11 

64.0000 

63.1191 

17.6608 

6.9137 

4 

7.66955 

48.0000 

26.1961 

12.3981 

-13.2608 

5 

22.0749 

10.0000 

26.1133 

14.5357 

-20.1464 

6 

40.0765 

8.0000 

9.2445 

8.7823 

-18.7051 

7 

88.4097 

6.5000 

13.4096 

11.7090 

-23.8543 

8 

101.506 

6.0000 

14.2004 

11.3907 

-22.0504 

9 

79.7902 

8.8000 

-0.3861 

15.1309 

-48.5401 

10 

19.8917 

• 

31.4122 

21.2165 

-36.1092 

11 

92.54 

• 

19.3664 

8.4602 

-7.5580 

UPPER95% 

LOWER954 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1. 

4 . 1398 

92.2472 

-43.5456 

115.5 

3.8E-12 

2 

29.2134 

86.6186 

-49.1201 

109.9 

0.5969 

3 

1057.11 

119.3 

-16.3969 

142.6 

0.8809 

4 

7.66955 

65.6530 

-42.5105 

94.9027 

21.8039 

5 

22.0749 

72.3730 

-46.7133 

98.9399 

-16.1133 

6 

40.0765 

37.1941 

-53.5642 

72.0532 

-1.2445 

7 

88.4097 

50.6734 

-54.0615 

80.8807 

-6.9096 

8 

101.506 

50.4513 

-52.7155 

81.1174 

-8.2004 

9 

79.7902 

47.7679 

-74.4304 

73.6582 

9.1861 

10 

19.8917 

98.9336 

-56.4678 

119,3 

« 

11 

92.54 

46.2909 

-42.9928 

81.7257 

• 

STD  ERR 

ID 

RESIDUAL 

1 

4.1398 

1.4E-05 

2 

29.2134 

0.5934 

3 

1057.11 

0.6812 

4 

7.66955 

12.5958 

5 

22.0749 

10.0540 

6 

40.0765 

15.3375 

7 

88.4097 

13.2388 

8 

101.506 

13.5137 

9 

79.7902 

9.1337 

10 

19.8917 

• 

11 

92.04 

• 

SUM  OF  RESIDUALS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


1.52323E-12 

937.1016 

637811.6 


E-81 


Regression  Model:  using  only  the  first  Principal  component 

of  the  highly  correlated  set  of  variables 

(x1 ,  x3 ,  x8,  X, ,  xa ,  x25) 


DEP  VARIABLE:  Y 

ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DF 

SQUARES 

SQUARE 

F  VALUE 

PROB>F 

MODEL 

.  1 

1477.14894 

1477.14894 

4.690 

0.0670 

ERROR 

7 

2204.55328 

314.93518 

C  TOTAL 

8 

3681.70222 

ROOT 

MSE 

17.74644 

R-SQUARE 

0.4012 

DEP  MEAN 

24.25556 

ADJ  R-SQ 

0.3157 

C.V. 

73.16444 

PARAMETER  ESTIMATES 


PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER" 0 

INTERCEP 

1 

17.88125956 

6.60725342 

2.706 

PI 

1 

0.04011814 

0.01852422 

2.166 

VARIABLE 

DF 

PROB  >  |*| 

INTERCEP 

1 

0.0304 

PI 

1 

0.0670 

OBS 

ID 

ACTUAL 

PREDICT 

VALUE 

STD  ERR 
PREDICT 

LOWERS 5% 
MEAN 

1 

4.1398 

36.0000 

18.0473 

6.5735 

2.5035 

2 

29.2134 

31.0000 

19.0532 

6.3846 

3.9559 

3 

1057.11 

64.0000 

60.2907 

17.6592 

18.5330 

4 

7.66955 

48.0000 

18.1889 

6.5452 

2.7119 

5 

22.0749 

10.0000 

18.7669 

6.4355 

3.5492 

6 

40.0765 

8.0000 

19.4891 

6.3116 

4.5643 

7 

88.4097 

6.5000  . 

21.4281 

6.0578 

7.1035 

8 

101.506 

6.0000 

.  21.9535 

6.0102 

7.7414 

9 

79.7902 

8.8000 

21.0823 

6.0942 

6.6716 

10 

19.8917 

• 

18.6793 

6.4516 

3.4237 

11 

92.54 

• 

21.5938 

6.0418 

7.3071 

E-B2 


OBS 


OBS 


UPPER9S% 

LOWERS 5% 

UPPER95% 

ID 

MEAN 

PREDICT 

PREDICT 

RESIDUAL 

1 

4.13S8 

33.5912 

-26.7030 

62.7976 

17.9527 

2 

29.2134 

34.1505 

-25.5439 

63.6504 

11.9468 

3 

1057.11 

102.0 

1.0903 

119.5 

3.7093 

4 

7.66055 

33.6660 

-26.5382 

62.9161 

29.8111 

5 

22.0749 

33.9846 

-25.8712 

63.4049 

-8.7669 

6 

40.0765 

34.4138 

-25.0500 

64.0281 

-11.4891 

7 

88.4097 

35.7527 

-22.9134 

65.7696 

-14.9281 

8 

101.506 

36.1655 

-22.3518 

66.2588 

-15.9535 

9 

79.7902 

35.4930 

-23.2871 

65.4517 

-12.2823 

10 

19.8917 

33.9349 

-25.9717 

63.3303 

• 

11 

92.54 

35.8805 

-22.7355 

65.9231 

• 

STD  ERR 

ID 

RESIDUAL 

1 

4.1398 

16.4841 

2 

29.2134 

16.5582 

3 

1057.11 

1.7577 

4 

7.66955 

16.4953 

5 

22.0749 

16.5384 

6 

40.0765 

16.5861 

7 

88.4097 

16.6805 

8 

101.506 

16.6977 

9 

79.7902  . 

16.6672 

10 

19.8917 

• 

11 

92.54 

• 

SUM  OF  RESIDUALS 

SUM  OF  SQUARED  RESIDUALS 

PREDICTED  RESID  SS  (PRESS) 


5. 24025E-14 
2204.553 
145859.2 


E-83 
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